# Technical Documentation: Michael-A-Kuykendall/shimmy > ℹ️ **Provenance:** Hybrid Fusion: `Michael-A-Kuykendall/shimmy` (README + 10 In-Tree Chapters) · [CodeWiki Reference](https://codewiki.google/github.com/Michael-A-Kuykendall/shimmy) · Recency: Active (< 180 days) ## 1. Project Overview & Quickstart (Michael-A-Kuykendall/shimmy) **Shimmy will be free forever.** No asterisks. No "free for now." No pivot to paid. ### 💝 Support Shimmy's Growth 🚀 **If Shimmy helps you, consider [sponsoring](https://github.com/sponsors/Michael-A-Kuykendall) — 100% of support goes to keeping it free forever.** - **$5/month**: Coffee tier ☕ - Eternal gratitude + sponsor badge - **$25/month**: Bug prioritizer 🐛 - Priority support + name in [SPONSORS.md](SPONSORS.md) - **$100/month**: Corporate backer 🏢 - Logo placement + monthly office hours - **$500/month**: Infrastructure partner 🚀 - Direct support + roadmap input [**🎯 Become a Sponsor**](https://github.com/sponsors/Michael-A-Kuykendall) | See our amazing [sponsors](SPONSORS.md) 🙏 --- ## What Is Shimmy? Shimmy is a **single-binary** that provides **100% OpenAI-compatible endpoints** for GGUF models. Point your existing AI tools to Shimmy and they just work — locally, privately, and free. Under the hood it runs on **Airframe**, a pure-Rust WebGPU (WGSL) transformer engine built from scratch. No C++ toolchain, no backend flags, no compilation required. Version history lives in the [CHANGELOG](CHANGELOG.md); see the [Airframe CHANGELOG](https://github.com/Michael-A-Kuykendall/airframe/blob/master/CHANGELOG.md) for engine release notes. **Why this matters:** - No C++ toolchain required — Rust only, top to bottom - F32 precision throughout for deterministic, high-quality output - WGSL compute shaders work on any GPU via WebGPU (NVIDIA, AMD, Intel, integrated) - Model spec auto-derived from GGUF metadata — no hardcoded per-model constants - YaRN RoPE scaling for extended context via `SHIMMY_MAX_CTX` (see [Extended Context](docs/EXTENDED_CONTEXT.md)) --- ## 🎯 Supported Models **11 model families · 25 certified model/quant combinations** — every model below passes Shimmy's 5-gate GPU math verification pipeline (dequant, structural peel, numerical, decode≡prefill, logits) against the certification ledger. GGUF files load as-is; no recompilation, no hardcoded per-model constants. | Family | Model | Quants | |---|---|---| | **Llama** | [Llama-3.2-1B-Instruct](https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF) | Q4_K_M · Q6_K | | | [Llama-3.2-3B-Instruct](https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF) | Q4_K_M | | | [TinyLlama-1.1B-Chat](https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF) | Q4_0 · Q5_K_M · Q6_K | | **Qwen3** | [Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B-GGUF) | Q4_K_M | | | [Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B-GGUF) | Q4_K_M | | | [Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B-GGUF) | Q4_K_M | | | [Qwen3-4B-Thinking](https://huggingface.co/Qwen/Qwen3-4B-Thinking-GGUF) | Q4_K_M | | | [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B-GGUF) | Q4_K_M | | **Qwen2** | [Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct-GGUF) | Q4_K_M | | | [Qwen2-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2-1.5B-Instruct-GGUF) | Q4_K_M | | | [Qwen2-7B-Instruct](https://huggingface.co/Qwen/Qwen2-7B-Instruct-GGUF) | Q4_K_M | | **Qwen3.5** | [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B-GGUF) | Q4_K_M | | **Phi-3** | [Phi-3.5-mini-Instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct-gguf) | Q4_K_M | | | [Phi-3-mini-4k-Instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf) | Q4_0 | | **Phi-2** | [Phi-2](https://huggingface.co/TheBloke/phi-2-GGUF) | Q4_K_M | | **Gemma-2** | [Gemma-2-2B-it](https://huggingface.co/bartowski/gemma-2-2b-it-GGUF) | Q4_K_M | | | [Gemma-2-9B-it](https://huggingface.co/bartowski/gemma-2-9b-it-GGUF) | Q4_K_M | | **Gemma-4** | [Gemma-4-12B-coder](https://huggingface.co/google/gemma-4-12B-coder-GGUF) | Q4_K_M | | | [Gemma-4-E4B](https://huggingface.co/google/gemma-4-E4B-it-GGUF) | Q4_K_M | | **DeepSeek-R1** | [DeepSeek-R1-0528-Qwen3-8B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-8B-GGUF) | Q4_K_M | | **Ministral** | [Ministral-3-14B-Reasoning](https://huggingface.co/bartowski/Ministral-3-14B-Reasoning-GGUF) | Q4_K_M | | **StarCoder2** | [StarCoder2-3B](https://huggingface.co/second-state/StarCoder2-3B-GGUF) | Q4_K_M | ## Features at a Glance - **⚡ [TurboShimmy INT4 KV Cache](docs/turboshimmy.md)** — ~7× less KV VRAM with one flag (`--kv-quant int4`). Run Llama-3.2-3B on 4 GB GPUs. - **🚀 [OpenAI SDK Compatibility](docs/INTEGRATION.md)** — drop-in replacement; VSCode Copilot, Cursor, Continue.dev, any OpenAI SDK. - **🔧 [Extended Context](docs/EXTENDED_CONTEXT.md)** — YaRN RoPE scaling via `SHIMMY_MAX_CTX`. - **📦 [Migrating from v1.x](docs/MIGRATION_v2.md)** — the llama.cpp backend was removed in v2.0; see the migration guide. - **🧠 MOE support** — Mixture-of-Experts CPU offloading is on the [Airframe roadmap](docs/AIRFRAME_MOE_ROADMAP.md). - **🏆 Certification** — Every model passes a 5-gate mathematical verification pipeline. See [docs/CERTIFICATION.md](docs/CERTIFICATION.md) for how it works. --- ## Quick Start (30 seconds) ```bash # 1) Download pre-built binary (Windows example) curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-windows-x86_64.exe -o shimmy.exe # 2) Point it at a GGUF model set SHIMMY_BASE_GGUF=C:\path\to\model.gguf && ./shimmy.exe serve & # 3) See registered models ./shimmy list # 4) Smoke test the OpenAI API curl -s http://127.0.0.1:11435/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"tinyllama-1.1b","messages":[{"role":"user","content":"Say hi in 5 words."}],"max_tokens":32}' \ | jq -r '.choices[0].message.content' ``` Full install, model acquisition, GPU, and VRAM sizing: **[docs/quickstart.md](docs/quickstart.md)** --- ## Documentation Hub Full documentation lives in [docs/](docs/). Use this table to find what you need: ### Getting Started | Document | Description | |---|---| | [quickstart.md](docs/quickstart.md) | Install, models, GPU, VRAM, extended context | | [MIGRATION_v2.md](docs/MIGRATION_v2.md) | Migrating from Shimmy v1.x | | [CONFIGURATION.md](docs/CONFIGURATION.md) | All environment variables and config options | | [WINDOWS_GPU_BUILD_GUIDE.md](docs/WINDOWS_GPU_BUILD_GUIDE.md) | Windows-specific build instructions | ### Models & Performance | Document | Description | |---|---| | [SUPPORTED_MODELS.md](docs/SUPPORTED_MODELS.md) | Certified models and quantization support | | [turboshimmy.md](docs/turboshimmy.md) | INT4 KV cache compression | | [EXTENDED_CONTEXT.md](docs/EXTENDED_CONTEXT.md) | YaRN RoPE scaling, VRAM math | | [MODEL_EXPANSION.md](docs/MODEL_EXPANSION.md) | Model onboarding protocol and acceptance gates | | [PERFORMANCE.md](docs/PERFORMANCE.md) | Performance tuning and token/sec benchmarks | ### API & Integration | Document | Description | |---|---| | [API.md](docs/API.md) | Complete endpoint, CLI, and env-var reference | | [OPENAI_COMPAT.md](docs/OPENAI_COMPAT.md) | OpenAI compatibility matrix — what's supported | | [INTEGRATION.md](docs/INTEGRATION.md) | LangChain, OpenAI SDKs, VSCode, etc. | | [EXAMPLES.md](docs/EXAMPLES.md) | Runnable code examples | | [CROSS_COMPILATION.md](docs/CROSS_COMPILATION.md) | Building for other targets (ARM, Linux from Windows) | ### Engine Deep Dives | Document | Description | |---|---| | [ARCHITECTURE.md](docs/ARCHITECTURE.md) | System-level architecture and component map | | [GPU_PIPELINE.md](docs/GPU_PIPELINE.md) | Bindless GPU architecture, WGSL shaders, dispatch patterns | | [QUANTIZATION.md](docs/QUANTIZATION.md) | Q4_0, Q8_0, K-quant formats — bit-level internals | | [CHAT_TEMPLATES.md](docs/CHAT_TEMPLATES.md) | Chat template auto-detection and format reference | ### FAQ & Troubleshooting | Document | Description | |---|---| | [FAQ.md](docs/FAQ.md) | Frequently asked questions | | [TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) | GPU errors, model failures, port conflicts | | [FEATURES.md](docs/FEATURES.md) | Complete feature list | ### Certification & Methodology | Document | Description | |---|---| | [CERTIFICATION.md](docs/CERTIFICATION.md) | How we mathematically prove every model is correct | | [METHODOLOGY.md](docs/METHODOLOGY.md) | Engineering methodology and quality standards | | [REGRESSION_TESTING.md](docs/REGRESSION_TESTING.md) | Regression testing approach | | [ppt-invariant-testing.md](docs/ppt-invariant-testing.md) | Property-based and invariant testing details | | [METRICS.md](docs/METRICS.md) | Observability and metrics reference | --- ## Development Testing Shimmy maintains high code quality through comprehensive testing: ```bash # Full test suite (default features = GPU engine) cargo test --features airframe,huggingface # Quick CPU-only tests (no GPU required) cargo test --lib --no-default-features --features huggingface -- --test-threads=1 ``` See [docs/ppt-invariant-testing.md](docs/ppt-invariant-testing.md) for technical details. --- ## Community & Support - **🐛 Bug Reports**: [GitHub Issues](https://github.com/Michael-A-Kuykendall/shimmy/issues) - **💬 Discussions**: [GitHub Discussions](https://github.com/Michael-A-Kuykendall/shimmy/discussions) - **💝 Sponsorship**: [GitHub Sponsors](https://github.com/sponsors/Michael-A-Kuykendall) ### Star History [](https://www.star-history.com/#Michael-A-Kuykendall/shimmy&Timeline) ### 🚀 Momentum Snapshot 🌟 ** stars and climbing fast** ⏱ **<1s startup** 🦀 **100% Rust, no Python** ### 📰 As Featured On 🔥 [**Hacker News**](https://news.ycombinator.com/item?id=45130322) • [**Front Page Again**](https://news.ycombinator.com/item?id=45199898) • [**IPE Newsletter**](https://ipenewsletter.substack.com/p/the-strange-new-side-hustles-of-openai) **Companies**: Need invoicing? Email [michaelallenkuykendall@gmail.com](mailto:michaelallenkuykendall@gmail.com) --- ## Performance Comparison | Tool | Startup Time | Memory Usage | OpenAI API | |------|--------------|--------------|------------| | **Shimmy** | **<100ms** | **50MB** | **100%** | | Ollama | 5-10s | 200MB+ | Partial | --- ## License & Philosophy MIT License - forever and always. **Philosophy**: Infrastructure should be invisible. Shimmy is infrastructure. **Testing Philosophy**: Reliability through comprehensive validation and property-based testing. --- **Forever maintainer**: Michael A. Kuykendall **Promise**: This will never become a paid product **Mission**: Making local model inference simple and reliable ## 2. In-Tree Documentation Chapters (Michael-A-Kuykendall/shimmy) ## File: README.md **Shimmy will be free forever.** No asterisks. No "free for now." No pivot to paid. ### 💝 Support Shimmy's Growth 🚀 **If Shimmy helps you, consider [sponsoring](https://github.com/sponsors/Michael-A-Kuykendall) — 100% of support goes to keeping it free forever.** - **$5/month**: Coffee tier ☕ - Eternal gratitude + sponsor badge - **$25/month**: Bug prioritizer 🐛 - Priority support + name in [SPONSORS.md](SPONSORS.md) - **$100/month**: Corporate backer 🏢 - Logo placement + monthly office hours - **$500/month**: Infrastructure partner 🚀 - Direct support + roadmap input [**🎯 Become a Sponsor**](https://github.com/sponsors/Michael-A-Kuykendall) | See our amazing [sponsors](SPONSORS.md) 🙏 --- ## What Is Shimmy? Shimmy is a **single-binary** that provides **100% OpenAI-compatible endpoints** for GGUF models. Point your existing AI tools to Shimmy and they just work — locally, privately, and free. Under the hood it runs on **Airframe**, a pure-Rust WebGPU (WGSL) transformer engine built from scratch. No C++ toolchain, no backend flags, no compilation required. Version history lives in the [CHANGELOG](CHANGELOG.md); see the [Airframe CHANGELOG](https://github.com/Michael-A-Kuykendall/airframe/blob/master/CHANGELOG.md) for engine release notes. **Why this matters:** - No C++ toolchain required — Rust only, top to bottom - F32 precision throughout for deterministic, high-quality output - WGSL compute shaders work on any GPU via WebGPU (NVIDIA, AMD, Intel, integrated) - Model spec auto-derived from GGUF metadata — no hardcoded per-model constants - YaRN RoPE scaling for extended context via `SHIMMY_MAX_CTX` (see [Extended Context](docs/EXTENDED_CONTEXT.md)) --- ## 🎯 Supported Models **11 model families · 25 certified model/quant combinations** — every model below passes Shimmy's 5-gate GPU math verification pipeline (dequant, structural peel, numerical, decode≡prefill, logits) against the certification ledger. GGUF files load as-is; no recompilation, no hardcoded per-model constants. | Family | Model | Quants | |---|---|---| | **Llama** | [Llama-3.2-1B-Instruct](https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF) | Q4_K_M · Q6_K | | | [Llama-3.2-3B-Instruct](https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF) | Q4_K_M | | | [TinyLlama-1.1B-Chat](https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF) | Q4_0 · Q5_K_M · Q6_K | | **Qwen3** | [Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B-GGUF) | Q4_K_M | | | [Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B-GGUF) | Q4_K_M | | | [Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B-GGUF) | Q4_K_M | | | [Qwen3-4B-Thinking](https://huggingface.co/Qwen/Qwen3-4B-Thinking-GGUF) | Q4_K_M | | | [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B-GGUF) | Q4_K_M | | **Qwen2** | [Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct-GGUF) | Q4_K_M | | | [Qwen2-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2-1.5B-Instruct-GGUF) | Q4_K_M | | | [Qwen2-7B-Instruct](https://huggingface.co/Qwen/Qwen2-7B-Instruct-GGUF) | Q4_K_M | | **Qwen3.5** | [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B-GGUF) | Q4_K_M | | **Phi-3** | [Phi-3.5-mini-Instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct-gguf) | Q4_K_M | | | [Phi-3-mini-4k-Instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf) | Q4_0 | | **Phi-2** | [Phi-2](https://huggingface.co/TheBloke/phi-2-GGUF) | Q4_K_M | | **Gemma-2** | [Gemma-2-2B-it](https://huggingface.co/bartowski/gemma-2-2b-it-GGUF) | Q4_K_M | | | [Gemma-2-9B-it](https://huggingface.co/bartowski/gemma-2-9b-it-GGUF) | Q4_K_M | | **Gemma-4** | [Gemma-4-12B-coder](https://huggingface.co/google/gemma-4-12B-coder-GGUF) | Q4_K_M | | | [Gemma-4-E4B](https://huggingface.co/google/gemma-4-E4B-it-GGUF) | Q4_K_M | | **DeepSeek-R1** | [DeepSeek-R1-0528-Qwen3-8B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-8B-GGUF) | Q4_K_M | | **Ministral** | [Ministral-3-14B-Reasoning](https://huggingface.co/bartowski/Ministral-3-14B-Reasoning-GGUF) | Q4_K_M | | **StarCoder2** | [StarCoder2-3B](https://huggingface.co/second-state/StarCoder2-3B-GGUF) | Q4_K_M | ## Features at a Glance - **⚡ [TurboShimmy INT4 KV Cache](docs/turboshimmy.md)** — ~7× less KV VRAM with one flag (`--kv-quant int4`). Run Llama-3.2-3B on 4 GB GPUs. - **🚀 [OpenAI SDK Compatibility](docs/INTEGRATION.md)** — drop-in replacement; VSCode Copilot, Cursor, Continue.dev, any OpenAI SDK. - **🔧 [Extended Context](docs/EXTENDED_CONTEXT.md)** — YaRN RoPE scaling via `SHIMMY_MAX_CTX`. - **📦 [Migrating from v1.x](docs/MIGRATION_v2.md)** — the llama.cpp backend was removed in v2.0; see the migration guide. - **🧠 MOE support** — Mixture-of-Experts CPU offloading is on the [Airframe roadmap](docs/AIRFRAME_MOE_ROADMAP.md). - **🏆 Certification** — Every model passes a 5-gate mathematical verification pipeline. See [docs/CERTIFICATION.md](docs/CERTIFICATION.md) for how it works. --- ## Quick Start (30 seconds) ```bash # 1) Download pre-built binary (Windows example) curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-windows-x86_64.exe -o shimmy.exe # 2) Point it at a GGUF model set SHIMMY_BASE_GGUF=C:\path\to\model.gguf && ./shimmy.exe serve & # 3) See registered models ./shimmy list # 4) Smoke test the OpenAI API curl -s http://127.0.0.1:11435/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"tinyllama-1.1b","messages":[{"role":"user","content":"Say hi in 5 words."}],"max_tokens":32}' \ | jq -r '.choices[0].message.content' ``` Full install, model acquisition, GPU, and VRAM sizing: **[docs/quickstart.md](docs/quickstart.md)** --- ## Documentation Hub Full documentation lives in [docs/](docs/). Use this table to find what you need: ### Getting Started | Document | Description | |---|---| | [quickstart.md](docs/quickstart.md) | Install, models, GPU, VRAM, extended context | | [MIGRATION_v2.md](docs/MIGRATION_v2.md) | Migrating from Shimmy v1.x | | [CONFIGURATION.md](docs/CONFIGURATION.md) | All environment variables and config options | | [WINDOWS_GPU_BUILD_GUIDE.md](docs/WINDOWS_GPU_BUILD_GUIDE.md) | Windows-specific build instructions | ### Models & Performance | Document | Description | |---|---| | [SUPPORTED_MODELS.md](docs/SUPPORTED_MODELS.md) | Certified models and quantization support | | [turboshimmy.md](docs/turboshimmy.md) | INT4 KV cache compression | | [EXTENDED_CONTEXT.md](docs/EXTENDED_CONTEXT.md) | YaRN RoPE scaling, VRAM math | | [MODEL_EXPANSION.md](docs/MODEL_EXPANSION.md) | Model onboarding protocol and acceptance gates | | [PERFORMANCE.md](docs/PERFORMANCE.md) | Performance tuning and token/sec benchmarks | ### API & Integration | Document | Description | |---|---| | [API.md](docs/API.md) | Complete endpoint, CLI, and env-var reference | | [OPENAI_COMPAT.md](docs/OPENAI_COMPAT.md) | OpenAI compatibility matrix — what's supported | | [INTEGRATION.md](docs/INTEGRATION.md) | LangChain, OpenAI SDKs, VSCode, etc. | | [EXAMPLES.md](docs/EXAMPLES.md) | Runnable code examples | | [CROSS_COMPILATION.md](docs/CROSS_COMPILATION.md) | Building for other targets (ARM, Linux from Windows) | ### Engine Deep Dives | Document | Description | |---|---| | [ARCHITECTURE.md](docs/ARCHITECTURE.md) | System-level architecture and component map | | [GPU_PIPELINE.md](docs/GPU_PIPELINE.md) | Bindless GPU architecture, WGSL shaders, dispatch patterns | | [QUANTIZATION.md](docs/QUANTIZATION.md) | Q4_0, Q8_0, K-quant formats — bit-level internals | | [CHAT_TEMPLATES.md](docs/CHAT_TEMPLATES.md) | Chat template auto-detection and format reference | ### FAQ & Troubleshooting | Document | Description | |---|---| | [FAQ.md](docs/FAQ.md) | Frequently asked questions | | [TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) | GPU errors, model failures, port conflicts | | [FEATURES.md](docs/FEATURES.md) | Complete feature list | ### Certification & Methodology | Document | Description | |---|---| | [CERTIFICATION.md](docs/CERTIFICATION.md) | How we mathematically prove every model is correct | | [METHODOLOGY.md](docs/METHODOLOGY.md) | Engineering methodology and quality standards | | [REGRESSION_TESTING.md](docs/REGRESSION_TESTING.md) | Regression testing approach | | [ppt-invariant-testing.md](docs/ppt-invariant-testing.md) | Property-based and invariant testing details | | [METRICS.md](docs/METRICS.md) | Observability and metrics reference | --- ## Development Testing Shimmy maintains high code quality through comprehensive testing: ```bash # Full test suite (default features = GPU engine) cargo test --features airframe,huggingface # Quick CPU-only tests (no GPU required) cargo test --lib --no-default-features --features huggingface -- --test-threads=1 ``` See [docs/ppt-invariant-testing.md](docs/ppt-invariant-testing.md) for technical details. --- ## Community & Support - **🐛 Bug Reports**: [GitHub Issues](https://github.com/Michael-A-Kuykendall/shimmy/issues) - **💬 Discussions**: [GitHub Discussions](https://github.com/Michael-A-Kuykendall/shimmy/discussions) - **💝 Sponsorship**: [GitHub Sponsors](https://github.com/sponsors/Michael-A-Kuykendall) ### Star History [](https://www.star-history.com/#Michael-A-Kuykendall/shimmy&Timeline) ### 🚀 Momentum Snapshot 🌟 ** stars and climbing fast** ⏱ **<1s startup** 🦀 **100% Rust, no Python** ### 📰 As Featured On 🔥 [**Hacker News**](https://news.ycombinator.com/item?id=45130322) • [**Front Page Again**](https://news.ycombinator.com/item?id=45199898) • [**IPE Newsletter**](https://ipenewsletter.substack.com/p/the-strange-new-side-hustles-of-openai) **Companies**: Need invoicing? Email [michaelallenkuykendall@gmail.com](mailto:michaelallenkuykendall@gmail.com) --- ## Performance Comparison | Tool | Startup Time | Memory Usage | OpenAI API | |------|--------------|--------------|------------| | **Shimmy** | **<100ms** | **50MB** | **100%** | | Ollama | 5-10s | 200MB+ | Partial | --- ## License & Philosophy MIT License - forever and always. **Philosophy**: Infrastructure should be invisible. Shimmy is infrastructure. **Testing Philosophy**: Reliability through comprehensive validation and property-based testing. --- **Forever maintainer**: Michael A. Kuykendall **Promise**: This will never become a paid product **Mission**: Making local model inference simple and reliable --- ## File: deploy/README.md # Shimmy Cloud Deployment One-click deployment configurations for popular cloud platforms. ## Quick Deploy Buttons ### Railway [](https://railway.app/template/shimmy) ### Render [](https://render.com/deploy) ### Fly.io ```bash # Install flyctl and deploy curl -L https://fly.io/install.sh | sh fly deploy ``` ### Docker (Any Platform) ```bash # Local development docker-compose up # Production with Nginx docker-compose --profile production up ``` ## Platform-Specific Instructions ### Railway.app 1. Click the "Deploy on Railway" button above 2. Connect your GitHub account 3. Fork this repository 4. Railway will automatically build and deploy 5. Your Shimmy instance will be available at `https://your-app.railway.app` ### Render.com 1. Click the "Deploy to Render" button above 2. Connect your GitHub repository 3. Render will use the `render.yaml` configuration 4. Your service will be available with automatic HTTPS ### Fly.io 1. Install the Fly CLI: `curl -L https://fly.io/install.sh | sh` 2. Clone this repository: `git clone https://github.com/Michael-A-Kuykendall/shimmy.git` 3. Navigate to the project: `cd shimmy` 4. Create and deploy: `fly deploy` 5. Access your app: `fly open` ### Google Cloud Run ```bash # Build and deploy to Cloud Run gcloud builds submit --tag gcr.io/PROJECT-ID/shimmy gcloud run deploy --image gcr.io/PROJECT-ID/shimmy --platform managed ``` ### AWS App Runner 1. Create `apprunner.yaml` in your repository root: ```yaml version: 1.0 runtime: docker build: commands: build: - echo "Building Shimmy with Docker" run: runtime-version: latest command: shimmy serve --bind 0.0.0.0:8080 network: port: 8080 ``` ### DigitalOcean App Platform 1. Create app via DigitalOcean control panel 2. Connect your GitHub repository 3. DigitalOcean will detect the Dockerfile automatically 4. Set environment variables as needed ## Environment Variables | Variable | Default | Description | |----------|---------|-------------| | `PORT` | `11434` | Port to bind the server | | `RUST_LOG` | `info` | Log level (error, warn, info, debug, trace) | | `SHIMMY_BIND` | `0.0.0.0:11434` | Full bind address | ## Resource Requirements ### Minimal - **CPU**: 0.5 vCPU - **Memory**: 512MB RAM - **Storage**: 100MB (binary only) ### Recommended - **CPU**: 1 vCPU - **Memory**: 1GB RAM - **Storage**: 1GB+ (for model caching) ### High Performance - **CPU**: 2+ vCPU - **Memory**: 4GB+ RAM - **Storage**: 10GB+ SSD ## Security Considerations 1. **Authentication**: Shimmy doesn't include built-in authentication. Use a reverse proxy (Nginx, Cloudflare) for auth. 2. **Rate Limiting**: The included Nginx configuration has basic rate limiting. Adjust as needed. 3. **HTTPS**: Most cloud platforms provide automatic HTTPS. For self-hosted deployments, configure SSL certificates. 4. **Firewall**: Only expose port 11434 (or your configured port) to the public internet. ## Monitoring ### Health Checks All configurations include health checks at `/health` endpoint. ### Logs Set `RUST_LOG=debug` for detailed logging. Most platforms provide log aggregation. ### Metrics For production deployments, consider adding: - Prometheus metrics - Jaeger tracing - Custom monitoring dashboards ## Scaling ### Horizontal Scaling Shimmy is stateless and can be horizontally scaled. Use a load balancer to distribute requests. ### Vertical Scaling For better performance with large models: - Increase memory for model caching - Add more CPU cores for parallel processing - Use SSD storage for faster model loading ## Troubleshooting ### Common Issues 1. **Out of Memory**: Increase memory allocation or use memory-mapped loading 2. **Slow Startup**: Enable model caching and use persistent storage 3. **Connection Timeout**: Increase proxy timeout settings for large model inference ### Debug Mode ```bash # Enable debug logging RUST_LOG=debug shimmy serve ``` ### Container Debugging ```bash # Access running container docker exec -it shimmy-container /bin/bash # Check logs docker logs shimmy-container ``` --- ## File: docs/benchmark-evidence/README.md # MoE CPU Offloading Benchmark Evidence <<<<<<< HEAD <<<<<<< HEAD **Date**: October 8, 2025 ======= **Date**: October 8, 2025 >>>>>>> main ======= **Date**: October 8, 2025 >>>>>>> main **Purpose**: Raw benchmark data and logs for audit verification ## Contents ### Streaming vs Non-Streaming Benchmarks - **phi35-streaming-bench.log** - Phi-3.5-MoE 41.9B performance comparison - **gpt-oss-streaming-bench.log** - GPT-OSS 20B performance comparison - **deepseek-streaming-bench.log** - DeepSeek MoE 16B performance comparison Each log contains: - 4 test prompts (short, medium, long, very long) - Non-streaming TPS measurements - Streaming TPS measurements with actual token counts - TTFT (Time To First Token) estimates - Performance delta calculations ### Model Loading and Offloading Logs - **shimmy-phi35.log** - Phi-3.5-MoE server startup with CPU offloading - **shimmy-gpt-oss.log** - GPT-OSS server startup with CPU offloading - **shimmy-deepseek.log** - DeepSeek server startup with CPU offloading Each log contains: - Model architecture detection (expert count, active experts) - Expert tensor CPU offloading confirmation - Memory distribution (GPU vs CPU allocation) - Context configuration ## Verification These logs provide evidence for claims in the MoE CPU Offloading White Paper: 1. **Expert Detection**: Search for `expert_count` and `expert_used_count` in loading logs 2. **CPU Offloading**: Search for `CUDA_Host` buffer overrides in loading logs 3. **Memory Savings**: Search for `CPU_Mapped` and `CUDA0 model buffer size` in loading logs 4. **Performance Data**: Raw TPS and TTFT measurements in streaming-bench logs ## Reproduction To reproduce these results: ```bash # Start shimmy server with CPU offloading cd /home/ubuntu/shimmy SHIMMY_BASE_GGUF=/path/to/model.gguf \ ./target/release/shimmy serve --bind 127.0.0.1:11435 --cpu-moe > server.log 2>&1 & # Run streaming benchmark ./scripts/benchmark-moe-streaming.sh > benchmark.log # Compare results with evidence files in this directory ``` ## File Integrity | File | Size | Date | Purpose | |------|------|------|---------| | phi35-streaming-bench.log | 2.6K | Oct 8, 2025 | Phi-3.5 benchmarks | | gpt-oss-streaming-bench.log | 2.6K | Oct 8, 2025 | GPT-OSS benchmarks | | deepseek-streaming-bench.log | 2.5K | Oct 8, 2025 | DeepSeek benchmarks | | shimmy-phi35.log | 414K | Oct 8, 2025 | Phi-3.5 loading logs | | shimmy-gpt-oss.log | 431K | Oct 8, 2025 | GPT-OSS loading logs | | shimmy-deepseek.log | 698K | Oct 8, 2025 | DeepSeek loading logs | --- *Evidence preserved for audit verification and reproducibility* --- ## File: docs/releases/RELEASE_NOTES_v1.6.0.md # Shimmy v1.6.0 - Windows CUDA Support + Critical Stability Fixes ## 🎯 Headline Features ### ✨ Native Windows CUDA Support **Shimmy is now the first lightweight Rust LLM tool with full Windows MSVC CUDA support.** - Fixed critical llama-cpp-rs Windows MSVC build issue via custom fork - CUDA binaries build successfully on Windows (24MB binary, 11m 25s compile time) - All 4 GPU backends now verified working on Windows: CUDA, Vulkan, OpenCL, HuggingFace - Fork available at: `Michael-A-Kuykendall/llama-cpp-rs` (branch: `fix-windows-msvc-cuda-stdbool`) **Technical Achievement:** Solved bindgen's inability to locate MSVC standard C headers by extracting paths from `cc` crate's environment detection and passing as `-isystem` arguments. This fix benefits the entire Rust + llama.cpp ecosystem. ### 🐛 Issue #72 Fixed: GPU Backend Flag Now Works Previous versions accepted the `--gpu-backend` flag but ignored it completely, assigning all layers to CPU. **What was broken:** - CLI parsed `--gpu-backend auto|vulkan|opencl|cuda` ✅ - Engine had `gpu_backend` field ✅ - **BUT:** Field was never used in model loading ❌ - **AND:** CLI value was never passed to engine constructor ❌ **What's fixed:** - Added `LlamaEngine::new_with_backend(Option<&str>)` constructor - Implemented auto-detection with priority: CUDA > Vulkan > OpenCL > CPU - GPU backend now properly wired through all CLI commands (serve, generate, gpu-info) - Verified with 13 comprehensive regression tests **User Impact:** Your GPU will actually be used now when you specify `--gpu-backend vulkan` 🎉 ## 🔧 Critical Stability Fixes ### Resolved RwLock Deadlock in Concurrent Operations Fixed an infinite hang when 20+ concurrent model loading operations occurred. **Root Cause:** `ModelManager::load_model()` held a write lock while calling async functions that tried to acquire read locks on the same data, creating a circular dependency. **Solution:** Extract data with locks, then drop locks before calling other functions. Simple pattern, massive impact. **Before:** `test_concurrent_load_unload` hung indefinitely **After:** Passes in 0.00s ### Fixed All Test Failures Across Feature Combinations - **295/295 tests passing** with full CUDA backend - **284/284 tests passing** with minimal features (huggingface only) - **295/295 tests passing** with any GPU backend (vulkan/opencl/cuda) **What we fixed:** 1. PPT contract tests now properly guarded by feature flags 2. Removed flaky property tests that used broken `property_test()` wrapper 3. Fixed `test_local_file_detection` to work without llama backend 4. All tests now deterministic, no random failures ## 📊 Build Verification | Backend | Build Time | Binary Size | Status | |---------|-----------|-------------|--------| | HuggingFace | 8s | 4.8MB | ✅ Pass | | Vulkan | 3m 19s | 4.8MB | ✅ Pass | | OpenCL | 45s | 4.8MB | ✅ Pass | | **CUDA** | **11m 25s** | **24MB** + 36MB lib | ✅ **Pass** | All 4 backends verified working on Windows with full test coverage. ## 🧹 Code Quality Improvements - Removed 700+ lines of dead code (unused specs, ModelCache, RouteManager) - Fixed all clippy warnings - Comprehensive audit and cleanup before release - Updated all "sub-20MB" references back to "sub-5MB" (actual: 4.8MB) - Removed AI marketing fluff from README ## 🔬 Technical Details ### Windows MSVC CUDA Fix Architecture The fix leverages `cc::Build` to extract MSVC's `INCLUDE` environment variable and passes those paths as `-isystem` arguments to bindgen's libclang: ```rust // Extract MSVC include paths from cc crate let include_paths = cc::Build::new() .target("x86_64-pc-windows-msvc") .get_compiler() .env() .iter() .filter(|(k, _)| k == "INCLUDE") .flat_map(|(_, v)| v.to_string_lossy().split(';').map(String::from).collect::>()) .collect::>(); // Pass to bindgen for path in include_paths { builder = builder.clang_arg(format!("-isystem{}", path)); } ``` Similar to the Android fix in llama-cpp-rs (lines 390-414), but adapted for MSVC's environment. ## 🚀 What's Next - Consider upstream PR to llama-cpp-rs with Windows MSVC fix - Continue monitoring regression gates for stability - Explore optimizations for faster CUDA build times ## 📦 Installation ```bash # Minimal build (HuggingFace only) cargo install shimmy --no-default-features --features huggingface # Vulkan (fastest compile, 45s) cargo install shimmy --no-default-features --features huggingface,llama-vulkan # OpenCL (AMD/Intel GPUs) cargo install shimmy --no-default-features --features huggingface,llama-opencl # CUDA (NVIDIA GPUs, longer compile) cargo install shimmy --no-default-features --features huggingface,llama-cuda ``` ## 🙏 Credits - Issue #72 reported by @D0wn10ad - Windows CUDA testing and validation by the community - llama-cpp-rs team for the excellent bindings foundation --- **Shimmy:** 4.8MB. No Python. No bloat. Now with Windows CUDA support. **142x smaller than Ollama. 2x faster model loading. 100% Rust.** --- ## File: docs/releases/RELEASE_NOTES_v1.7.0.md # Shimmy v1.7.0 - Mixture of Experts CPU Offloading Release <<<<<<< HEAD **Released:** January 9, 2025 ======= **Released:** January 9, 2025 >>>>>>> main **Branch:** `feat/moe-cpu-offload` --- ## 🎯 Headline Features ### Mixture of Experts (MoE) CPU Offloading Support **Major new capability enabling large MoE models on consumer GPUs** - requested by [@razvanab](https://github.com/razvanab) in [Issue #81](https://github.com/Michael-A-Kuykendall/shimmy/issues/81). This release adds full support for offloading Mixture of Experts (MoE) model weights to CPU memory, dramatically reducing VRAM requirements while maintaining usable inference performance. Now you can run massive models like **GPT-OSS 20B**, **Phi-3.5-MoE 42B**, and **DeepSeek-16B** on GPUs with limited VRAM. **New CLI Flags:** - `--cpu-moe` - Offload all MoE expert tensors to CPU memory - `--n-cpu-moe N` - Offload N expert layers to CPU (partial offloading) **Performance Achievements:** - **78%-94% VRAM reduction** across tested models - **2.5x-6.9x speed penalty** (acceptable for development/prototyping) - Successfully validated on **Lambda Cloud GH200 (96GB VRAM)** **Example Usage:** ```bash # Full CPU offload (maximum VRAM savings) shimmy serve --cpu-moe --model gpt-oss-20b.gguf # Partial offload (balance VRAM vs speed) shimmy serve --n-cpu-moe 64 --model phi-3.5-moe.gguf # Generate with offloading shimmy generate --cpu-moe --model deepseek-16b.gguf --prompt "Hello" ``` **Technical Implementation:** - Rust bindings to llama.cpp's MoE offloading functionality via `llama-cpp-2` fork - Integration through engine adapter with global CLI flags - Verified with 144 expert tensors successfully offloaded on GPT-OSS 20B - Comprehensive testing: 36/36 test runs passing (3 models × 2 configs × 3 runs × 2 quantizations) --- ## 📦 Quantized Models Released Six professionally quantized MoE models uploaded to HuggingFace with comprehensive model cards following bartowski/Microsoft standards: ### Phi-3.5-MoE Quantizations (from 79GB F16) #### 1. Q2_K - Ultra-Compressed (15GB, 81% reduction) - **Repository:** [MikeKuykendall/phi-3.5-moe-q2-k-cpu-offload-gguf](https://huggingface.co/MikeKuykendall/phi-3.5-moe-q2-k-cpu-offload-gguf) - **Direct Download:** [phi-3.5-moe-q2-k-cpu-offload.gguf](https://huggingface.co/MikeKuykendall/phi-3.5-moe-q2-k-cpu-offload-gguf/resolve/main/phi-3.5-moe-q2-k-cpu-offload.gguf) (15.0 GB) - **Use Case:** Maximum compression, development/testing, low VRAM systems - **Quality:** Acceptable for most tasks, noticeable quality loss vs F16 #### 2. Q4_K_M - Recommended (24GB, 70% reduction) - **Repository:** [MikeKuykendall/phi-3.5-moe-q4-k-m-cpu-offload-gguf](https://huggingface.co/MikeKuykendall/phi-3.5-moe-q4-k-m-cpu-offload-gguf) - **Direct Download:** [phi-3.5-moe-q4-k-m-cpu-offload.gguf](https://huggingface.co/MikeKuykendall/phi-3.5-moe-q4-k-m-cpu-offload-gguf/resolve/main/phi-3.5-moe-q4-k-m-cpu-offload.gguf) (23.8 GB) - **Use Case:** Best quality/size balance, general production use - **Quality:** Minimal quality loss vs F16, recommended for most users #### 3. Q8_0 - High Quality (42GB, 47% reduction) - **Repository:** [MikeKuykendall/phi-3.5-moe-q8-0-cpu-offload-gguf](https://huggingface.co/MikeKuykendall/phi-3.5-moe-q8-0-cpu-offload-gguf) - **Direct Download:** [phi-3.5-moe-q8-0-cpu-offload.gguf](https://huggingface.co/MikeKuykendall/phi-3.5-moe-q8-0-cpu-offload-gguf/resolve/main/phi-3.5-moe-q8-0-cpu-offload.gguf) (41.7 GB) - **Use Case:** Maximum quality, near F16 performance - **Quality:** Virtually identical to F16 ### DeepSeek-MoE-16B Quantizations (from 31GB F16) #### 4. Q2_K - Ultra-Compressed (6.3GB, 80% reduction) - **Repository:** [MikeKuykendall/deepseek-moe-16b-q2-k-cpu-offload-gguf](https://huggingface.co/MikeKuykendall/deepseek-moe-16b-q2-k-cpu-offload-gguf) - **Direct Download:** [deepseek-moe-16b-q2-k-cpu-offload.gguf](https://huggingface.co/MikeKuykendall/deepseek-moe-16b-q2-k-cpu-offload-gguf/resolve/main/deepseek-moe-16b-q2-k-cpu-offload.gguf) (6.32 GB) - **Use Case:** Maximum compression, development/testing - **Quality:** Acceptable for most tasks, noticeable quality loss vs F16 #### 5. Q4_K_M - Recommended (11GB, 65% reduction) - **Repository:** [MikeKuykendall/deepseek-moe-16b-q4-k-m-cpu-offload-gguf](https://huggingface.co/MikeKuykendall/deepseek-moe-16b-q4-k-m-cpu-offload-gguf) - **Direct Download:** [deepseek-moe-16b-q4-k-m-cpu-offload.gguf](https://huggingface.co/MikeKuykendall/deepseek-moe-16b-q4-k-m-cpu-offload-gguf/resolve/main/deepseek-moe-16b-q4-k-m-cpu-offload.gguf) (10.9 GB) - **Use Case:** Best quality/size balance, general production use - **Quality:** Minimal quality loss vs F16, recommended for most users #### 6. Q8_0 - High Quality (17GB, 45% reduction) - **Repository:** [MikeKuykendall/deepseek-moe-16b-q8-0-cpu-offload-gguf](https://huggingface.co/MikeKuykendall/deepseek-moe-16b-q8-0-cpu-offload-gguf) - **Direct Download:** [deepseek-moe-16b-q8-0-cpu-offload.gguf](https://huggingface.co/MikeKuykendall/deepseek-moe-16b-q8-0-cpu-offload-gguf/resolve/main/deepseek-moe-16b-q8-0-cpu-offload.gguf) (16.7 GB) - **Use Case:** Maximum quality, near F16 performance - **Quality:** Virtually identical to F16 **Model Card Features:** - Proper YAML metadata (language, license, tags, base_model, pipeline_tag) - Real performance benchmarks from controlled A/B testing - VRAM usage with/without CPU offloading - Token generation speeds (TPS) with detailed methodology - Usage examples for shimmy CLI integration - Quantization methodology and technical specifications ### Complete Model Comparison Table | Model | Quantization | Size | Reduction vs F16 | Download URL | Use Case | |-------|--------------|------|------------------|--------------|----------| | **Phi-3.5-MoE** (79GB F16) | | | | | | | | Q2_K | 15.0 GB | 81% | [Download](https://huggingface.co/MikeKuykendall/phi-3.5-moe-q2-k-cpu-offload-gguf/resolve/main/phi-3.5-moe-q2-k-cpu-offload.gguf) | Maximum compression | | | Q4_K_M ⭐ | 23.8 GB | 70% | [Download](https://huggingface.co/MikeKuykendall/phi-3.5-moe-q4-k-m-cpu-offload-gguf/resolve/main/phi-3.5-moe-q4-k-m-cpu-offload.gguf) | **Recommended** | | | Q8_0 | 41.7 GB | 47% | [Download](https://huggingface.co/MikeKuykendall/phi-3.5-moe-q8-0-cpu-offload-gguf/resolve/main/phi-3.5-moe-q8-0-cpu-offload.gguf) | Maximum quality | | **DeepSeek-16B** (31GB F16) | | | | | | | | Q2_K | 6.32 GB | 80% | [Download](https://huggingface.co/MikeKuykendall/deepseek-moe-16b-q2-k-cpu-offload-gguf/resolve/main/deepseek-moe-16b-q2-k-cpu-offload.gguf) | Maximum compression | | | Q4_K_M ⭐ | 10.9 GB | 65% | [Download](https://huggingface.co/MikeKuykendall/deepseek-moe-16b-q4-k-m-cpu-offload-gguf/resolve/main/deepseek-moe-16b-q4-k-m-cpu-offload.gguf) | **Recommended** | | | Q8_0 | 16.7 GB | 45% | [Download](https://huggingface.co/MikeKuykendall/deepseek-moe-16b-q8-0-cpu-offload-gguf/resolve/main/deepseek-moe-16b-q8-0-cpu-offload.gguf) | Maximum quality | ⭐ = Recommended quantization level for production use **Testing Validation:** - **36 baseline tests** completed (100% success rate) - **N=3 statistical runs** per configuration for reliability - **Controlled A/B comparisons** (with/without `--cpu-moe`) - **Lambda Cloud GH200** infrastructure (96GB VRAM, 72 CPU cores) - **shimmy v1.6.0** used for all test runs --- ## 🔧 Technical Details ### Upstream Contributions **llama-cpp-rs Fork Integration:** - Using custom fork: `utilityai/llama-cpp-rs` (branch: `feat/moe-cpu-offload`) - Added Rust bindings: `with_cpu_moe_all()`, `with_n_cpu_moe(n)` methods - Submitted upstream PR: [utilityai/llama-cpp-rs#839](https://github.com/utilityai/llama-cpp-rs/pull/839) (CUDA stdbool fix) - Clean integration via Cargo dependency override in Cargo.toml **Implementation Architecture:** ``` CLI Flags (--cpu-moe, --n-cpu-moe) ↓ Global Config (MoeConfig struct) ↓ Engine Adapter (apply_moe_config) ↓ llama-cpp-2 Bindings (LlamaParams) ↓ llama.cpp MoE Offloading (native C++) ``` ### Performance Benchmarks **Phi-3.5-MoE Q4_K_M (24GB model):** - Baseline (no offload): 11.55 TPS, ~23GB VRAM - With `--cpu-moe`: 4.69 TPS, ~2MB VRAM (2.5x speed penalty, 99.9% VRAM reduction) **GPT-OSS 20B Q8_0 (17GB model):** - Baseline (no offload): 12.3 TPS, ~15GB VRAM - With `--cpu-moe`: 1.78 TPS, ~2MB VRAM (6.9x speed penalty, 99.9% VRAM reduction) **DeepSeek-16B Q8_0 (17GB model):** - Baseline (no offload): 14.2 TPS, ~16GB VRAM - With `--cpu-moe`: 3.1 TPS, ~2MB VRAM (4.6x speed penalty, 99.9% VRAM reduction) **TTFT (Time to First Token):** - Minimal impact: <500ms increase with CPU offloading - Dominated by model loading, not offloading configuration ### Code Quality Improvements **Systematic Audit Cleanup (Phases 1-3):** - **Phase 1 (I2 Pattern):** Renamed 22 Java-style getters to Rust conventions - `get_model()` → `model()`, `get_metrics()` → `metrics()`, etc. - All call sites updated, 295/295 tests passing <<<<<<< HEAD ======= >>>>>>> main - **Phase 2 (N5 Pattern):** Fixed 14 production unwraps with proper error handling - `src/metrics.rs` (5 unwraps), `src/openai_compat.rs` (3 unwraps) - Replaced with `match`, `unwrap_or_else`, `unwrap_or` patterns - 226+ test unwraps remain (acceptable - tests should panic) - **Phase 3 (A3_stringly Pattern):** Converted 16+ string errors to typed ShimmyError - New variants: `WorkflowStepNotFound`, `MlxNotAvailable`, `ToolExecutionFailed`, etc. - Typed errors in `workflow.rs`, `safetensors_adapter.rs`, `tools.rs`, `preloading.rs` - Engine layer kept with `anyhow::Result` (clean boundary for third-party errors) **Build Verification:** - All 295 unit tests passing - Zero compiler warnings (achieved clean build) - Clippy clean (removed unnecessary conversions, unused imports) - Formatting verified with `cargo fmt` ### Startup Diagnostics Enhancement **New Serve Command Output:** ``` 🚀 Shimmy v1.7.0 🖥️ Backend: CUDA (GPU acceleration enabled) 🧠 MoE: CPU offload enabled (all experts) 📚 Models: 0 available 🌐 Starting server on 127.0.0.1:11435 📚 Models: 3 available ✅ Ready to serve requests • POST /api/generate (streaming + non-streaming) • GET /health (health check + metrics) • GET /v1/models (OpenAI-compatible) ``` **Benefits:** - Immediate configuration feedback before first request - GPU backend visibility (CPU/CUDA/Vulkan/OpenCL/auto-detected) - MoE config shown at startup (when feature enabled) - Model discovery progress (shows count twice: before/after scan) - Error prevention (wrong config visible instantly) **Implementation:** - Zero performance overhead (<1ms) - Works with `RUST_LOG=off` (uses stdout) - Emoji markers for visual scanning - 7 new unit tests, 204/204 bin tests passing --- ## 🐛 Critical Fixes ### Issue #85: Template Compilation Errors in crates.io Installation **Problem:** `cargo install shimmy` failed with template generation errors - Nested tokio runtime panics during template file generation - Async functions causing runtime conflicts **Solution:** - Remove async from template generation functions (they were synchronous) - Eliminate nested tokio runtime causing panics - Template files properly included in package, runtime issue was the blocker **Verification:** - Fresh install from crates.io: `cargo install shimmy --features llama` - Template generation working correctly - All integration tests passing ### Issue #84: Startup Diagnostics Implementation **Problem:** No visibility into shimmy configuration until first request fails - Wrong GPU backend only discovered after server starts - Missing MoE config not shown until generation attempted - No model count feedback during discovery **Solution:** Added comprehensive startup diagnostics (see Technical Details above) **Testing:** - Manual testing on Windows with CUDA - 7 new unit tests for diagnostic output formatting - Regression tests: 204/204 bin tests, 295/295 lib tests passing ### MoE Config Application Fix **Problem:** `--cpu-moe` flags ignored when auto-registering discovered models in serve command **Root Cause:** Serve command created new LlamaEngine without MoE configuration **Solution:** - Apply MoE config to both initial engine AND enhanced_engine - Ensure expert tensor offloading works in serve mode - Verified: 144 expert tensors offloaded to CPU with GPT-OSS 20B model **Testing:** - Manual verification with GPT-OSS 20B (144 experts offloaded) - Phi-3.5-MoE and DeepSeek-16B validation - All serve mode configurations tested --- ## 📚 Documentation Updates ### HuggingFace Model Cards **Professional Standards:** - All 6 model cards follow bartowski/Microsoft style - Real performance benchmarks (not estimates) - Comprehensive YAML metadata (language, license, tags, base_model, pipeline_tag) - Usage examples with shimmy CLI integration - Quantization methodology and technical specifications **Metadata Audit & Corrections:** - Fixed "empty or missing yaml metadata" warnings - Corrected DeepSeek base_model references (was pointing to wrong model) - All repos rendering correctly on HuggingFace - Proper tag relationships (GGUF, quantized, transformers) ### Internal Documentation Organization **Moved to `docs/internal/`:** - `EXECUTION-PLAN-QUANTIZATION-TO-HF.md` - `MODEL-CARD-PLAN.md` - `MOE-TESTING-STATUS.md` - `QUANTIZATION-PERFORMANCE-SUMMARY.md` - `QUANTIZATION-STATUS-REPORT.md` - `QUANTIZATION-TESTING-PLAN.md` - `QUANTIZATION-UPLOAD-COMPLETE.md` - `UPLOAD-COMMANDS.md` - `HUGGINGFACE-AUDIT-2025-10-09.md` **Benefits:** - Cleaner repository root - Internal planning docs separated from user-facing documentation - Historical context preserved for future development --- ## 🔮 What's Next ### Planned Enhancements - **Additional quantization levels:** Q3_K_M, Q5_K_M for quality/size balance - **More MoE models:** Qwen-3-235B, Mixtral variants with CPU offloading - **Benchmark suite:** Automated A/B testing framework for MoE configs - **Dynamic offloading:** Runtime adjustment of expert tensor placement - **VRAM monitoring:** Real-time VRAM usage tracking during inference ### Community Contributions - Upstream PR pending: [utilityai/llama-cpp-rs#839](https://github.com/utilityai/llama-cpp-rs/pull/839) - Testing feedback welcome on Issue #81 - Additional model requests via GitHub issues --- ## 📥 Installation ### From Source (Recommended for MoE Support) ```bash git clone https://github.com/Michael-A-Kuykendall/shimmy.git cd shimmy git checkout feat/moe-cpu-offload cargo build --release --features llama ./target/release/shimmy --version ``` ### From crates.io (Standard Features) ```bash cargo install shimmy --features llama shimmy --version ``` ### Quick Start with MoE Models ```bash # Example 1: Phi-3.5-MoE Q4_K_M (Recommended - Best Balance) # Download the model (24GB) wget https://huggingface.co/MikeKuykendall/phi-3.5-moe-q4-k-m-cpu-offload-gguf/resolve/main/phi-3.5-moe-q4-k-m-cpu-offload.gguf \ -O phi-3.5-moe-q4-k-m.gguf # Run with CPU offloading shimmy serve --cpu-moe --model phi-3.5-moe-q4-k-m.gguf # Test generation curl -X POST http://localhost:11435/api/generate \ -H "Content-Type: application/json" \ -d '{"prompt": "Explain quantum computing in simple terms", "max_tokens": 100}' # Example 2: DeepSeek-16B Q2_K (Smallest - Maximum VRAM Savings) # Download the model (6.3GB) wget https://huggingface.co/MikeKuykendall/deepseek-moe-16b-q2-k-cpu-offload-gguf/resolve/main/deepseek-moe-16b-q2-k-cpu-offload.gguf \ -O deepseek-moe-16b-q2-k.gguf # Run with CPU offloading shimmy serve --cpu-moe --model deepseek-moe-16b-q2-k.gguf # Example 3: Phi-3.5-MoE Q8_0 (Highest Quality - Near F16) # Download the model (42GB) wget https://huggingface.co/MikeKuykendall/phi-3.5-moe-q8-0-cpu-offload-gguf/resolve/main/phi-3.5-moe-q8-0-cpu-offload.gguf \ -O phi-3.5-moe-q8-0.gguf # Run with partial CPU offloading (64 layers) shimmy serve --n-cpu-moe 64 --model phi-3.5-moe-q8-0.gguf # Example 4: Using huggingface-cli (Alternative Download Method) # Install: pip install huggingface-hub huggingface-cli download MikeKuykendall/phi-3.5-moe-q4-k-m-cpu-offload-gguf \ phi-3.5-moe-q4-k-m-cpu-offload.gguf --local-dir ./models shimmy serve --cpu-moe --model ./models/phi-3.5-moe-q4-k-m-cpu-offload.gguf ``` ### Quantization Selection Guide **Choose Q2_K if:** - You have very limited disk space (<10GB available) - You're doing rapid prototyping/testing - Quality is less critical than VRAM savings - You want the absolute smallest model size **Choose Q4_K_M if (RECOMMENDED):** - You want the best balance of quality and size - You're deploying to production - You need reliable performance across diverse tasks - You have 12-30GB disk space available **Choose Q8_0 if:** - You need maximum quality (virtually identical to F16) - You have sufficient disk space (17-42GB) - You're doing critical work requiring best possible output - You can afford slightly larger VRAM usage --- ## 🙏 Credits **Special Thanks:** - **[@razvanab](https://github.com/razvanab)** for suggesting MoE CPU offloading in [Issue #81](https://github.com/Michael-A-Kuykendall/shimmy/issues/81) - this entire release exists because of your feature request! 🎉 - **Lambda Labs** for providing GH200 GPU infrastructure for comprehensive testing - **llama.cpp team** for the upstream MoE offloading implementation - **bartowski** for setting the standard with professional HuggingFace model cards **Contributors:** - Michael A. Kuykendall ([@Michael-A-Kuykendall](https://github.com/Michael-A-Kuykendall)) - Lead development, quantization, testing - Claude Code (Anthropic) - Code refactoring assistance, documentation --- ## 🔗 Related Links - **Issue #81:** [Feature Request - MoE CPU Offloading](https://github.com/Michael-A-Kuykendall/shimmy/issues/81) - **Issue #84:** [Startup Diagnostics](https://github.com/Michael-A-Kuykendall/shimmy/issues/84) - **Issue #85:** [Template Compilation Fix](https://github.com/Michael-A-Kuykendall/shimmy/issues/85) - **PR #839:** [llama-cpp-rs CUDA stdbool Fix](https://github.com/utilityai/llama-cpp-rs/pull/839) - **HuggingFace Models:** [MikeKuykendall Profile](https://huggingface.co/MikeKuykendall) - **Previous Release:** [v1.6.0 Release Notes](./RELEASE_NOTES_v1.6.0.md) --- ## 📊 Detailed Changelog ### New Features - `--cpu-moe` flag for full MoE CPU offloading - `--n-cpu-moe N` flag for partial MoE CPU offloading - Startup diagnostics with GPU backend and MoE config visibility - 6 quantized MoE models on HuggingFace with professional documentation ### Bug Fixes - Fixed `--cpu-moe` flags being ignored in serve command - Resolved template compilation errors in crates.io installation - Fixed ANSI color output (respects NO_COLOR and TERM env vars) - Corrected HuggingFace model card metadata (YAML, base_model references) ### Code Quality - Renamed 22 Java-style getters to Rust conventions (I2 pattern) - Fixed 14 production unwraps with proper error handling (N5 pattern) - Converted 16+ string errors to typed ShimmyError (A3_stringly pattern) - Achieved zero compiler warnings and clean clippy output ### Documentation - 6 professional HuggingFace model cards with real benchmarks - Organized 9 internal planning docs into `docs/internal/` - Created comprehensive v1.7.0 release notes - Updated copilot instructions with audit progress ### Testing - 36/36 quantization baseline tests passing (N=3 statistical runs) - 295/295 unit tests passing - 204/204 bin tests passing - Validated on Lambda Cloud GH200 (96GB VRAM, 72 cores) ### Infrastructure - Lambda Cloud GH200 testing environment - HuggingFace integration for model distribution - Custom llama-cpp-rs fork with MoE bindings - Cargo dependency override for upstream contributions --- **Full Changelog:** https://github.com/Michael-A-Kuykendall/shimmy/compare/v1.6.0...feat/moe-cpu-offload --- ## File: docs/releases/RELEASE_PREP_V1.7.2.md # Release Preparation Report - v1.7.2 **Generated:** October 9, 2025 **Branch:** `feature/mlx-native-support` **Target Version:** v1.7.2 **Previous Version:** v1.7.1 --- ## ✅ Merge Status ### PR #97 Merged Successfully - **PR:** feat(moe): complete v1.7.0 MOE CPU offloading implementation - **Commits:** 8 squashed commits - **Status:** ✅ Merged into `main` at commit `67bb6af` - **Content:** Complete MOE CPU offloading with 3 validated models ### ✅ MOE Work Integrated into MLX Branch - **Merge Commit:** `c488b59` - **Branch State:** `feature/mlx-native-support` now has both MLX + MOE work - **Commits Ahead:** 9 commits ahead of `origin/main` --- ## 📦 What's New in v1.7.2 ### 1. MOE CPU Offloading (from PR #97) - **CLI Flags:** - `--cpu-moe`: Offload ALL expert tensors to CPU - `--n-cpu-moe N`: Offload first N expert layers to CPU - **Performance Metrics:** - GPT-OSS 20B: 71.5% VRAM reduction, 6.9x speed penalty - Phi-3.5-MoE 42B: 99.9% VRAM reduction, 2.5x speed penalty - DeepSeek MoE 16B: 99.9% VRAM reduction, 4.6x speed penalty - **Testing:** 36 test result files, N=3 statistical validation - **Documentation:** Complete whitepapers, technical reports, model cards ### 2. MLX Apple Silicon Support (new) - **Platform:** Native Apple Silicon (M-series) support - **Backend:** MLX engine for Metal GPU acceleration - **Feature Flag:** `--features mlx` and `apple` feature set - **Files:** - `src/engine/mlx.rs`: MLX engine implementation - `.github/workflows/mlx-apple-silicon.yml`: CI workflow (✅ passing) - `tests/mlx_support_regression_test.rs`: Regression tests - `MLX_IMPLEMENTATION_PLAN.md`: Implementation documentation - **Status:** ✅ Workflow passing on GitHub Actions (macos-14 runner) - **Fix:** Resolved `--bin shimmy` binary ambiguity issue ### 3. Infrastructure Improvements - Pre-commit hooks disabled (preventing workflow friction) - Release gates system integrated - Comprehensive regression test suite - GitHub Actions workflow improvements --- ## 🧪 Release Gates Checklist ### Pre-Release Validation #### Compilation & Build - [ ] `cargo build --release` (no warnings) - [ ] `cargo build --release --all-features` (all features compile) - [ ] `cargo build --release --no-default-features --features llama` (llama only) - [ ] `cargo build --release --no-default-features --features mlx` (MLX only) - [ ] `cargo build --release --no-default-features --features apple` (Apple set) - [ ] `cargo build --release --no-default-features --features gpu` (Windows GPU) #### Testing - [ ] `cargo test --all-features` (all tests pass) - [ ] `cargo test --features mlx mlx` (MLX-specific tests) - [ ] `tests/mlx_support_regression_test.rs` (Issue #68 regression) - [ ] `tests/packaging_regression_test.rs` (crates.io package) - [ ] `tests/template_compilation_regression_test.rs` (template files) - [ ] `tests/apple_silicon_detection_test.rs` (GPU detection) - [ ] `tests/release_gate_integration.rs` (release gates) #### Lint & Format - [ ] `cargo fmt -- --check` (formatting) - [ ] `cargo clippy --all-features` (no warnings) - [ ] `cargo deny check` (license/dependency check) #### Documentation - [ ] README.md updated with new features - [ ] CHANGELOG.md has v1.7.2 entry - [ ] All new features documented in docs/ - [ ] Model cards updated (if applicable) #### Regression Tests (User-Reported Issues) - [ ] Issue #68: MLX support in macOS binaries - [ ] Issue #80: LLM model filtering in discovery - [ ] Issue #81: MOE CPU offloading - [ ] Issue #83: Template compilation in crates.io - [ ] Issue #87: Apple Silicon GPU detection - [ ] Issue #92: Bind address 'auto' panic #### GitHub Actions - [ ] CI workflow passing - [ ] MLX Apple Silicon workflow passing - [ ] Release workflow configured - [ ] All checks green before merge --- ## 🔍 Known Issues & Risks ### Resolved - ✅ Pre-commit hooks causing merge friction → Disabled - ✅ MLX workflow binary ambiguity → Fixed with `--bin shimmy` - ✅ PR #97 merge conflicts → Resolved and merged ### Outstanding - ⚠️ Need to verify no compilation warnings - ⚠️ Need to run full regression test suite - ⚠️ Need to update version number in Cargo.toml (currently 1.7.1) --- ## 📝 Release Steps ### 1. Pre-Release Testing (NOW) ```bash # Build all feature combinations cargo build --release --all-features cargo build --release --no-default-features --features apple cargo build --release --no-default-features --features gpu # Run full test suite cargo test --all-features cargo test --features mlx mlx # Check for warnings cargo clippy --all-features -- -D warnings # Format check cargo fmt -- --check ``` ### 2. Version Bump ```bash # Update Cargo.toml version to 1.7.2 # Update CHANGELOG.md with release notes # Commit version bump git commit -am "chore: bump version to 1.7.2" ``` ### 3. Merge to Main ```bash git checkout main git merge feature/mlx-native-support git push origin main ``` ### 4. Tag Release ```bash git tag -a v1.7.2 -m "Release v1.7.2: MLX Apple Silicon + MOE CPU Offloading" git push origin v1.7.2 ``` ### 5. GitHub Release - Create release from tag v1.7.2 - Upload release binaries (from CI) - Include release notes from CHANGELOG --- ## 📊 Commits Since v1.7.1 ``` c488b59 merge: integrate MOE CPU offloading from main into MLX branch 88e1878 chore: stage MLX branch changes before merge 29c1e40 fix(mlx): specify --bin shimmy in workflow to resolve binary ambiguity a352af6 fix MLX workflow Python environment issue a41631b remove pre-commit hooks causing issues 6ca1d00 fix: make release gates OS-agnostic and fix fragile pre-commit hooks 0b0e8e2 feat(mlx): implement native Apple Silicon MLX support with pre-commit quality gates 67bb6af feat(moe): complete v1.7.0 MOE CPU offloading implementation (#97) ``` **Total:** 8 new commits for v1.7.2 --- ## 🎯 Next Actions 1. **RUN RELEASE GATES** - Execute full test suite 2. **CHECK COMPILATION WARNINGS** - Ensure clean build 3. **UPDATE VERSION** - Bump to 1.7.2 in Cargo.toml 4. **UPDATE CHANGELOG** - Document all changes 5. **MERGE TO MAIN** - Final integration 6. **TAG & RELEASE** - Create v1.7.2 release --- **Status:** Ready for release gates testing **Blocker:** None (all merges complete) **Risk Level:** Low (both features tested independently) --- ## File: docs/releases/release-notes-v1.7.0.md # 🚀 Shimmy v1.7.0: The MoE Revolution is Here! ## 💥 BREAKTHROUGH: Run 42B+ Models on Consumer Hardware **Shimmy v1.7.0** unleashes the **MoE (Mixture of Experts) CPU Offloading Revolution** - enabling massive expert models to run on everyday GPUs with **up to 99.9% VRAM reduction**. --- ## 🔥 What's New & Game-Changing ### ⚡ MoE CPU Offloading Technology Transform impossible into possible: <<<<<<< HEAD - **`--cpu-moe`**: Automatically offload MoE layers to CPU ======= - **`--cpu-moe`**: Automatically offload MoE layers to CPU >>>>>>> main - **`--n-cpu-moe N`**: Fine-tune performance with precise layer control - **Massive Memory Savings**: 15GB models → 4GB VRAM usage - **Enterprise Ready**: Deploy 42B parameter models on 8GB consumer cards ### 📊 Real Performance Gains (Validated) - **GPT-OSS 20B**: 71.5% VRAM reduction (15GB → 4.3GB actual measurement) - **Phi-3.5-MoE 42B**: Runs on consumer hardware for the first time - **DeepSeek 16B**: Intelligent CPU-GPU hybrid execution - **Smart Tradeoffs**: Accept 2-7x slower inference for 10-100x memory savings ### 🛠️ Technical Excellence - **First-Class Rust**: Enhanced llama.cpp bindings with MoE support - **Cross-Platform**: Windows MSVC CUDA, macOS ARM64 Metal, Linux x86_64/ARM64 - **Production Tested**: 295/295 tests passing, comprehensive validation pipeline - **Still Tiny**: Sub-5MB binary maintains legendary efficiency --- ## 🎯 Use Cases Unlocked ### 🏢 Enterprise Deployment - **Cost Revolution**: Run large models without GPU farm investments <<<<<<< HEAD - **Scalable AI**: Deploy expert models on existing infrastructure - **Flexible Performance**: Balance speed vs. memory for any workload - **On-Premises Ready**: Keep sensitive data in-house with minimal hardware ### 🔬 Research & Development ======= - **Scalable AI**: Deploy expert models on existing infrastructure - **Flexible Performance**: Balance speed vs. memory for any workload - **On-Premises Ready**: Keep sensitive data in-house with minimal hardware ### 🔬 Research & Development >>>>>>> main - **Democratized Access**: Test large models on developer laptops - **Rapid Iteration**: Prototype MoE architectures efficiently - **Educational Power**: Advanced AI models accessible to everyone - **Hybrid Intelligence**: Combine CPU and GPU resources intelligently --- ## 🚀 Quick Start Your MoE Journey ### Installation Options ```bash # Install from crates.io (LIVE NOW!) cargo install shimmy # Or grab platform binaries below ⬇️ ``` ### 🤖 Ready-to-Use MoE Models **Curated collection on HuggingFace - optimized for CPU offloading:** #### 🥇 **Recommended Starting Points** ```bash # Download and run Phi-3.5-MoE 42B (Q4 K-M) - Best balance of quality/performance huggingface-cli download MikeKuykendall/phi-3.5-moe-q4-k-m-cpu-offload-gguf ./shimmy serve --cpu-moe --model-path phi-3.5-moe-q4-k-m.gguf # Or DeepSeek-MoE 16B (Q4 K-M) - Faster alternative huggingface-cli download MikeKuykendall/deepseek-moe-16b-q4-k-m-cpu-offload-gguf ./shimmy serve --cpu-moe --model-path deepseek-moe-16b-q4-k-m.gguf ``` #### 📊 **Complete Model Collection** | Model | Size | Quantization | VRAM | Use Case | Download | |-------|------|--------------|------|----------|----------| | **Phi-3.5-MoE** | 42B | Q8.0 | ~4GB | 🏆 Maximum Quality | [`phi-3.5-moe-q8-0-cpu-offload-gguf`](https://huggingface.co/MikeKuykendall/phi-3.5-moe-q8-0-cpu-offload-gguf) | | **Phi-3.5-MoE** | 42B | Q4 K-M | ~2.5GB | ⚡ **Recommended** | [`phi-3.5-moe-q4-k-m-cpu-offload-gguf`](https://huggingface.co/MikeKuykendall/phi-3.5-moe-q4-k-m-cpu-offload-gguf) | | **Phi-3.5-MoE** | 42B | Q2 K | ~1.5GB | 🚀 Ultra Fast | [`phi-3.5-moe-q2-k-cpu-offload-gguf`](https://huggingface.co/MikeKuykendall/phi-3.5-moe-q2-k-cpu-offload-gguf) | | **DeepSeek-MoE** | 16B | Q8.0 | ~2GB | 🎯 High Precision | [`deepseek-moe-16b-q8-0-cpu-offload-gguf`](https://huggingface.co/MikeKuykendall/deepseek-moe-16b-q8-0-cpu-offload-gguf) | | **DeepSeek-MoE** | 16B | Q4 K-M | ~1.2GB | ⭐ **Budget Pick** | [`deepseek-moe-16b-q4-k-m-cpu-offload-gguf`](https://huggingface.co/MikeKuykendall/deepseek-moe-16b-q4-k-m-cpu-offload-gguf) | | **DeepSeek-MoE** | 16B | Q2 K | ~800MB | 💨 Lightning Fast | [`deepseek-moe-16b-q2-k-cpu-offload-gguf`](https://huggingface.co/MikeKuykendall/deepseek-moe-16b-q2-k-cpu-offload-gguf) | | **GPT-OSS** | 21B | Various | ~3GB | 🔬 Research/Testing | [`gpt-oss-20b-moe-cpu-offload-gguf`](https://huggingface.co/MikeKuykendall/gpt-oss-20b-moe-cpu-offload-gguf) | #### 🎯 **Model Selection Guide** - **🥇 First Time?** → Phi-3.5-MoE Q4 K-M (best balance) <<<<<<< HEAD - **💪 High-End GPU (8GB+)?** → Phi-3.5-MoE Q8.0 (maximum quality) ======= - **💪 High-End GPU (8GB+)?** → Phi-3.5-MoE Q8.0 (maximum quality) >>>>>>> main - **💻 Limited VRAM (4GB)?** → DeepSeek-MoE Q4 K-M (budget friendly) - **⚡ Speed Critical?** → DeepSeek-MoE Q2 K (blazing fast) - **🔬 Research/Validation?** → GPT-OSS 21B (proven baseline) ### ⚡ Launch Commands ```bash # Enable MoE CPU offloading magic ./shimmy serve --cpu-moe --port 11435 --model-path your-model.gguf # Fine-tune performance for your hardware ./shimmy serve --n-cpu-moe 8 --port 11435 --model-path your-model.gguf # Standard OpenAI-compatible API - zero changes to your code! curl -X POST http://localhost:11435/v1/completions \ -H "Content-Type: application/json" \ -d '{"model": "your-model", "prompt": "Explain quantum computing in simple terms"}' ``` --- ## 📦 Cross-Platform Binaries **Choose your platform and start the revolution:** | Platform | Binary | Features | |----------|--------|----------| | 🐧 **Linux x86_64** | `shimmy-linux-x86_64` | SafeTensors + llama.cpp + MoE | | 🦾 **Linux ARM64** | `shimmy-linux-arm64` | Native ARM64 + full MoE support | | 🪟 **Windows x86_64** | `shimmy-windows-x86_64.exe` | CUDA GPU + MoE offloading | | 🍎 **macOS Intel** | `shimmy-macos-intel` | SafeTensors + Apple MLX | | 🚀 **macOS Apple Silicon** | `shimmy-macos-arm64` | Metal GPU + MLX + MoE power | All binaries include **zero Python dependencies** and **native SafeTensors support**. --- ## 🌟 Why This Changes Everything Before Shimmy v1.7.0: *"I need a $10,000 GPU to run expert models"* After Shimmy v1.7.0: *"I'm running 42B models on my gaming laptop"* This isn't just an update - it's **sustainable AI democratization**. Organizations can now: - ✅ Deploy cutting-edge models without infrastructure overhaul - ✅ Experiment with state-of-the-art architectures on existing hardware - ✅ Scale AI capabilities based on actual needs, not hardware limits - ✅ Maintain complete data sovereignty with on-premises deployment --- ## 📈 Validated & Transparent - **Multi-Model Testing**: 3 models validated across all platforms <<<<<<< HEAD - **Real Baselines**: Controlled A/B testing with actual measurements ======= - **Real Baselines**: Controlled A/B testing with actual measurements >>>>>>> main - **Production Quality**: Comprehensive release gate system - **Open Development**: [Technical validation report](docs/MOE-TECHNICAL-VALIDATION.md) available --- ## 🤝 Join the Revolution - **🚀 Start Now**: `cargo install shimmy` - **📚 Learn More**: [Technical Documentation](docs/) - **🐛 Report Issues**: [GitHub Issues](https://github.com/Michael-A-Kuykendall/shimmy/issues) - **🔗 Upstream**: Supporting [llama-cpp-rs PR #839](https://github.com/utilityai/llama-cpp-rs/pull/839) --- <<<<<<< HEAD **Ready to revolutionize your AI deployment?** The future of efficient model serving is here. Download Shimmy v1.7.0 and experience the MoE revolution! 🚀 ======= **Ready to revolutionize your AI deployment?** The future of efficient model serving is here. Download Shimmy v1.7.0 and experience the MoE revolution! 🚀 >>>>>>> main --- ## File: docs/zh-CN/CHAT_TEMPLATES.md # 对话模板参考 本文档说明 Shimmy 如何将对话消息格式化为模型输入，介绍三大模板族的格式细节，并解释自动识别逻辑——帮助你在输出异常时快速定位问题。 --- ## 为什么模板很重要 大型语言模型在训练时使用了特定的**提示格式**。当输入格式与训练时不匹配，模型会产生奇怪的行为——输出提前截断、重复原始输入内容，或者持续生成无意义的文本。 这是实际工作中**最常见的静默失败场景**。模型运行正常，API 返回 200，但输出明显不对。 --- ## 三大模板族 ### 1. ChatML **使用此模板的模型**：TinyLlama、Phi-2、StarCoder2（已通过验证），以及大多数未被识别为 Llama-3 的模型（默认回退）。 **格式：** ``` <|im_start|>system {system_message}<|im_end|> <|im_start|>user {user_message}<|im_end|> <|im_start|>assistant ``` **停止 token**：`<|im_end|>`、`<|im_start|>` **完整示例：** ``` <|im_start|>system 你是一个有帮助的助手。<|im_end|> <|im_start|>user 中国的首都是哪里？<|im_end|> <|im_start|>assistant 中国的首都是北京。<|im_end|> ``` --- ### 2. Llama-3 **使用此模板的模型**：Llama-3.2-1B-Instruct、Llama-3.2-3B-Instruct，以及所有 Meta Llama-3 系列。 **格式：** ``` <|begin_of_text|><|start_header_id|>system<|end_header_id|> {system_message}<|eot_id|><|start_header_id|>user<|end_header_id|> {user_message}<|eot_id|><|start_header_id|>assistant<|end_header_id|> ``` **停止 token**：`<|eot_id|>`、`<|end_of_text|>` **关键细节**：`<|end_header_id|>` 后面有两个换行符，这是正确格式的必要组成部分。停止 token 必须是 `<|eot_id|>`——不是 `\n`，不是 `###`，不是 `<|im_end|>`。 --- ### 3. OpenChat **使用此模板的模型**：专门采用 OpenChat 训练格式的模型。 **格式：** ``` GPT4 Correct User: {user_message}<|end_of_turn|>GPT4 Correct Assistant: ``` **停止 token**：此模板无额外停止 token（使用模型默认的 EOS token）。 --- ## 自动识别逻辑 Shimmy 根据模型文件名自动推断模板类型，规则如下（按优先级排序）： | 模型名称包含 | 识别为 | |------------|--------| | `llama-3`、`llama3`、`meta-llama-3` | Llama-3 | | 其他所有情况 | ChatML（默认） | **实现细节**：识别逻辑在 `infer_template()` 函数中，对模型名称进行小写比较，因此大小写不影响识别结果。 **当自动识别失败时**：可通过 API 请求中的 `template` 字段手动指定： ```json { "model": "local", "template": "chatml", "messages": [...] } ``` 可选值：`"chatml"`、`"llama3"`、`"openchat"`。 --- ## 各模型模板对照 | 模型 | 模板 | 停止 Token | |------|------|-----------| | TinyLlama-1.1B-Chat | ChatML | `<\|im_end\|>` | | Llama-3.2-1B-Instruct | Llama-3 | `<\|eot_id\|>` | | Llama-3.2-3B-Instruct | Llama-3 | `<\|eot_id\|>` | | Phi-2 | ChatML | `<\|im_end\|>` | | StarCoder2-3B | ChatML\* | 建议使用 `/v1/completions` | | GPT-2 | 无（补全模型） | 建议使用 `/v1/completions` | \* StarCoder2 是代码补全模型，不是指令模型。虽然可以接受 ChatML 格式输入，但建议直接使用 `/v1/completions` 接口效果更好。 --- ## 补全接口 vs 对话接口 Shimmy 同时支持两个接口： **`/v1/chat/completions`**（对话格式）： ```json { "model": "local", "messages": [ {"role": "system", "content": "你是代码助手"}, {"role": "user", "content": "写一个快速排序"} ] } ``` **`/v1/completions`**（原始补全格式，适合代码模型）： ```json { "model": "local", "prompt": "def quicksort(arr):", "max_tokens": 200, "temperature": 0.2 } ``` 对于 StarCoder2 和 GPT-2，优先使用 `/v1/completions`——它们在补全任务上的效果远好于模拟对话格式。 --- ## 自定义停止 Token 通过 API 可以添加额外的停止条件： ```json { "model": "local", "messages": [...], "stop": ["###", "---", "\n\n"] } ``` **注意事项**： - 停止 token 必须是词表中的**单个 token**，不能是多 token 序列 - 字符串 `"###"` 如果对应多个 token 则不会生效 - 建议使用模型原生的停止 token（如上表中的格式） --- ## 调试模板问题 **常见症状与原因：** | 症状 | 可能原因 | |------|---------| | 输出包含原始模板标签（如 `<\|im_start\|>`） | 模型用了错误的模板，停止 token 未生效 | | 回复重复用户输入 | 对话模板格式错误 | | 生成停在奇怪的地方 | 错误的停止 token 被触发 | | 每次输出内容相同 | 对话历史未正确传递 | ```bash # 查看原始请求的模板渲染结果 RUST_LOG=debug shimmy serve 2>&1 | grep -i template ``` --- ## 延伸阅读 - [故障排查指南](TROUBLESHOOTING.md) — 生成质量问题的系统排查 - [量化格式详解](QUANTIZATION.md) — 模型格式与兼容性 - [GPU 推理管线](GPU_PIPELINE.md) — Token 采样链路 --- > 💝 **如果 Shimmy 对您有帮助，欢迎[赞助支持](https://github.com/sponsors/Michael-A-Kuykendall)——所有款项 100% 用于保持项目永久免费。** --- ## File: docs/zh-CN/EXTENDED_CONTEXT.md # 扩展上下文窗口 本文档介绍 Airframe 如何通过 **YaRN RoPE 缩放**突破模型原生上下文长度限制——包含数学原理、各模型的显存计算方法，以及在不同硬件上的实际配置建议。 --- ## 什么是上下文窗口？ 上下文窗口是模型在生成响应时能够"看到"的最大 token 数量。超出这个限制，模型要么截断输入，要么产生无意义的输出——它根本无法引用窗口之外的内容。 常见模型的原生上下文长度： | 模型 | 原生上下文 | |------|-----------| | TinyLlama 1.1B | 2,048 | | Phi-2 2.7B | 2,048 | | Llama-3.2-1B / 3B | 131,072 | | Llama-3.2-3B | 131,072 | | StarCoder2 3B | 16,384 | | Gemma-2 2B | 8,192 | --- ## RoPE 位置编码原理 Transformer 模型通过**旋转位置编码（RoPE）**感知 token 在序列中的位置。对于序列位置 `t`、注意力头维度中的第 `i` 个维度： $$\theta(t, i) = \frac{t}{\text{base}^{2i / d_{\text{head}}}}$$ 其中 `base` 通常为 10000，`d_head` 为注意力头的维度。 每个 Q 和 K 向量按此角度旋转，注意力得分的计算自然地编码了 token 之间的相对距离。 **问题所在**：模型训练时只见过 `t ≤ native_ctx` 的位置。当 `t` 超出训练范围，旋转角度进入模型从未见过的区域，输出质量迅速下降。 --- ## YaRN：上下文长度扩展方案 **YaRN（Yet another RoPE extensioN）**通过对位置序列进行缩放，将超出范围的位置映射回模型熟悉的区域： $$\theta_{\text{YaRN}}(t, i) = \frac{t / s}{\text{base}^{2i / d_{\text{head}}}}$$ 缩放因子 `s` 的计算公式： $$s = \frac{\text{max\_ctx}}{\text{native\_ctx}}$$ **触发条件**：当 `SHIMMY_MAX_CTX > native_ctx` 时，Airframe 自动启用 YaRN。无需手动配置——设置环境变量即可： ```bash SHIMMY_MAX_CTX=8192 ./shimmy serve ``` 若 `SHIMMY_MAX_CTX ≤ native_ctx`，RoPE 按原始参数运行，不引入任何额外开销。 --- ## 显存（VRAM）计算 KV 缓存是上下文扩展的显存瓶颈。计算公式： $$\text{KV 缓存（字节）} = n_{\text{layers}} \times n_{\text{kv\_heads}} \times d_{\text{head}} \times \text{max\_ctx} \times 2 \times 4$$ 各模型实际用量： ### TinyLlama 1.1B （22 层，4 个 KV 头，head\_dim = 64） | 上下文长度 | KV 缓存大小 | |-----------|------------| | 2,048（原生） | 88 MB | | 4,096 | 176 MB | | 8,192 | 352 MB | ### Llama-3.2-1B （16 层，8 个 KV 头，head\_dim = 64） | 上下文长度 | KV 缓存大小 | |-----------|------------| | 8,192 | 512 MB | | 16,384 | 1,024 MB | | 32,768 | 2,048 MB | ### Llama-3.2-3B （28 层，8 个 KV 头，head\_dim = 128） | 上下文长度 | KV 缓存大小 | |-----------|------------| | 8,192 | 1,792 MB | | 16,384 | 3,584 MB | > **注意**：上述数值仅为 KV 缓存。总显存需求还需加上模型权重本身。 --- ## 按显存容量的推荐配置 | 显存 | 推荐模型 + 上下文 | |------|-----------------| | 4 GB | TinyLlama Q4_0，最高 4K 上下文 | | 6 GB | Llama-3.2-1B Q4_K_M，最高 8K 上下文 | | 8 GB | Llama-3.2-3B Q4_K_M，最高 4K 上下文 | | 12 GB | Llama-3.2-3B Q4_K_M，最高 8K 上下文 | | 16 GB | 7B 模型 Q4_K_M，最高 8K 上下文 | | 集成显卡（共享内存） | TinyLlama Q4_0，最高 2K 上下文 | --- ## YaRN 质量说明 YaRN 是一种近似方法，不能完全还原模型在超长上下文上的性能： - **1× 以内**（`max_ctx ≤ native_ctx`）：完整训练精度，无损失 - **1–2×**（略超原生上下文）：质量轻微下降，日常任务基本无感 - **2–4×**：远端内容的注意力质量逐渐下降，长文档理解可能受影响 - **4× 以上**：输出质量显著退化，仅适合实验性用途 对于 TinyLlama（原生 2K），`SHIMMY_MAX_CTX=8192` 是一个合理的实用上限。 --- ## 针脚测试（Needle-in-a-Haystack）基准结果 以下为 RTX 3060 12GB 上的实测结果（TinyLlama 1.1B Q4_0）： | 上下文长度 | 注入深度 15% | 注入深度 50% | 注入深度 85% | |-----------|------------|------------|------------| | 2,048 | ✅ 找到 | ✅ 找到 | ✅ 找到 | | 4,096 | ✅ 找到 | ✅ 找到 | ⚠️ 偶发漏检 | | 8,192 | ✅ 找到 | ⚠️ 偶发漏检 | ⚠️ 不稳定 | "注入深度"指目标信息在整个上下文中的位置百分比。头部和尾部内容的检索准确率通常高于中间位置。 --- ## 环境变量配置 | 变量 | 说明 | 默认值 | |------|------|--------| | `SHIMMY_MAX_CTX` | 覆盖最大上下文长度 | 读取模型 GGUF 文件中的原生值 | | `LIBSHIMMY_MODEL_PATH` | 模型文件路径（服务器模式） | 无 | | `SHIMMY_BASE_GGUF` | 模型文件路径（CLI 模式） | 自动发现 | --- ## 延伸阅读 - [量化格式详解](QUANTIZATION.md) — 压缩格式与显存占用 - [故障排查指南](TROUBLESHOOTING.md) — 上下文溢出与 OOM 错误处理 - [GPU 推理管线](GPU_PIPELINE.md) — KV 缓存在 GPU 内存中的布局 --- > 💝 **如果 Shimmy 对您有帮助，欢迎[赞助支持](https://github.com/sponsors/Michael-A-Kuykendall)——所有款项 100% 用于保持项目永久免费。** --- ## File: docs/zh-CN/GPU_PIPELINE.md # GPU 推理管线 本文档深入解析 Airframe 在 GPU 上运行 Transformer 推理的完整过程——着色器调度架构、无绑定资源模型、KV 缓存管理，以及采样链路。 适合阅读对象：准备修改着色器的贡献者、排查 GPU 层面故障的开发者，以及希望理解底层机制的用户。 --- ## 架构总览 ``` HTTP 请求抵达服务器 │ ▼ shimmy openai_compat 层 - 解析 JSON 请求 - 应用对话模板 → 提示词字符串 - 构建 SamplingParams（温度、top_p、惩罚系数、停止 token） │ ▼ airframe runtime::gpu::GpuRuntime::generate() - Tokenize（分词）提示词 - 预填充阶段：处理提示词 token → 填充 KV 缓存 - 解码阶段：逐 token 自回归生成 - 反 Tokenize → 响应字符串 │ ▼ shimmy → HTTP 响应 ``` --- ## 无绑定资源模型 传统 WebGPU 需要将每个缓冲区单独绑定到绑定槽。大型 Transformer 模型有数千个张量——逐一绑定会触及 WebGPU 的绑定数量上限，并在每次调度时产生巨大开销。 Airframe 采用**无绑定（Bindless）设计**：每层的权重张量打包进一个大型存储缓冲区，WGSL 着色器通过推送常量（WebGPU 中以 uniform 缓冲区实现）中的字节偏移量来索引： ``` 层缓冲区布局（每个 Transformer 层一个）： ┌────────────────────────────────────────────────────────────┐ │ attn_norm_weight │ attn_q.weight │ attn_k.weight │ │ │ attn_v.weight │ attn_o.weight │ ffn_norm_weight │ │ │ ffn_gate.weight │ ffn_up.weight │ ffn_down.weight │ │ └────────────────────────────────────────────────────────────┘ ↑ 每个张量区域以 GGUF 量化格式存储。 着色器通过元数据缓冲区获取各区域的字节偏移量。 ``` **调试提示**：如遇缓冲区绑定上限或绑定组创建失败的错误，问题通常不在无绑定权重缓冲区本身，而是 KV 缓存或激活缓冲区（这些仍独立绑定）。 --- ## 预填充阶段 — 分块处理 **预填充阶段**一次性处理完整的输入提示词，填充 KV 缓存。这是单次请求中 GPU 计算最密集的部分。 Airframe 将长提示词切分为 **512 token 的分块**，以避免 GPU 命令编码器超时： ``` 提示词 = 2048 token │ ┌────────▼────────┐ │ 分块 1：512 │ → KV 缓存 0..511 ← 前向传播，写入 KV │ 分块 2：512 │ → KV 缓存 512..1023 │ 分块 3：512 │ → KV 缓存 1024..1535 │ 分块 4：512 │ → KV 缓存 1536..2047 └─────────────────┘ │ 只使用最后一个分块的 logits 来生成第一个采样 token。 所有分块的 KV 状态保留在 GPU 缓冲区中。 ``` **为什么是 512？** WebGPU 对 GPU 命令执行时间有上限限制，超时会被操作系统终止。512 token 的分块在所有已测试硬件上（RTX 3060 到集成显卡）都能安全完成。 **调试跟踪**：设置 `AIRFRAME_TRACE_PREFILL_CHUNKS=1` 可以记录分块边界和处理时间。 --- ## 解码阶段 — 自回归生成 预填充完成后，解码器逐 token 生成输出。每个解码步骤： 1. **嵌入**：通过嵌入查找表将上一个 token 转换为向量 2. **前向传播**：穿过所有 Transformer 层（使用 KV 缓存） - 注意力机制使用 KV 缓存——当前 Q 关注所有过去的 K/V 对 - 每步只需计算位置 `t` 的新 Q/K/V 向量 3. **输出映射**：将 LM Head 应用到激活值，得到词表上的 logits 4. **采样**：从 logits 分布中选择下一个 token 5. **检查停止条件**：EOS token、额外停止 token、max_tokens 上限 6. 以新 token 返回第 1 步重复 每个解码步骤需要**完整的一次模型前向传播**。对于 22 层的 TinyLlama，每生成一个 token 就要执行 22 次注意力计算 + 22 次 FFN 计算。 **解码是内存带宽瓶颈，不是算力瓶颈**。速度限制来自从显存读取权重张量，而非浮点计算能力。显存带宽更大的 GPU 生成速度更快。 --- ## Transformer 层计算（WGSL 着色器解析） 每个 Transformer 层按以下顺序执行： ``` 输入激活值（形状：[seq_len, n_embd]） │ ├── RMS Norm → normalized_x（稳定数值） │ ├── Q 投影：normalized_x × attn_q.weight → Q [seq, n_heads, head_dim] ├── K 投影：normalized_x × attn_k.weight → K [seq, n_kv_heads, head_dim] ├── V 投影：normalized_x × attn_v.weight → V [seq, n_kv_heads, head_dim] │ ├── RoPE：对 Q 和 K 应用旋转位置编码 │ θ(t, i) = (t / yarn_scale) / base^(2i / head_dim) │ YaRN：当 ctx > native_ctx 时 yarn_scale = max_ctx / native_ctx │ ├── 将 K、V 写入位置 t 的 KV 缓存 │ ├── 注意力得分：Q × Kᵀ / √head_dim → scores [seq, n_heads, seq] ├── Softmax（在 seq 维度） → weights ├── 加权求和：weights × V → attn_out [seq, n_heads, head_dim] │ ├── 输出投影：attn_out × attn_o.weight → residual_add │ ├── 残差连接：x = x + residual_add │ ├── RMS Norm → normalized_for_ffn │ ├── FFN gate：normalized_for_ffn × ffn_gate.weight → gate ├── FFN up： normalized_for_ffn × ffn_up.weight → up ├── SwiGLU 激活：gate × sigmoid(gate) × up ├── FFN down：swiglu_out × ffn_down.weight → ffn_residual │ └── 残差连接：x = x + ffn_residual → 输出激活值 ``` **反量化在矩阵乘法内部实时进行**：每个矩阵乘法着色器在读取量化权重块时立即解码，然后与激活值相乘，结果直接累加到输出，不产生额外的显存分配。 --- ## KV 缓存 KV 缓存存储过去的键值对，避免在每个解码步骤重新计算。 **缓冲区布局：** ``` key_cache: [n_layers][n_kv_heads][max_ctx][head_dim] f32 value_cache: [n_layers][n_kv_heads][max_ctx][head_dim] f32 ``` 以 F32 格式存储为扁平 GPU 缓冲区。总大小： ``` key_cache_bytes = n_layers × n_kv_heads × max_ctx × head_dim × 4 total = key_cache_bytes × 2 ``` TinyLlama @ 2048 上下文：`22 × 4 × 2048 × 64 × 2 × 4 ≈ 88 MB` **写入**：预填充和每个解码步骤中，当前位置的新 K/V 向量被写入 `cache[layer][head][position]`。 **读取**：注意力计算时，读回到 `current_position` 为止的完整 K/V 切片。 **重置**：每次请求之间缓存会被重置（清零所有位置）。Shimmy 是**无状态**的——不维护会话级别的 KV 缓存。 --- ## 采样链路 前向传播产生词表上的 logits（长度为 `vocab_size` 的浮点向量）后，采样按以下顺序处理： ``` logits[vocab_size] │ 1. 重复惩罚（如果 repeat_penalty > 1.0）： 对过去 N 个 token 中出现过的每个 token t： logits[t] /= repeat_penalty（提高重复的"代价"） │ 2. 温度缩放： logits[i] = logits[i] / temperature （temperature = 0.0 → 贪心；temperature → ∞ → 均匀随机） │ 3. Softmax： probs[i] = exp(logits[i]) / Σ exp(logits[j]) │ 4. Top-p（核采样）： 按概率降序排列。 累计到总和 ≥ top_p 阈值为止。 对保留的 token 重新归一化。 从此压缩分布中采样。 │ 5. 采样 → token_id ``` **确定性输出（贪心解码）**：`temperature=0.0, top_p=1.0` **创意生成**：`temperature=0.8, top_p=0.95` 重复惩罚与 API 的 `frequency_penalty` / `presence_penalty` 字段的映射关系： ``` raw = max(frequency_penalty, presence_penalty) 如果 raw > 0.0：repeat_penalty = 1.0 + raw × 0.5 ``` 采样器在 CPU 上运行（logits 在前向传播后从 GPU 显存读回）。对于词表规模 32K 以内的模型，每 token 添加的延迟 < 1ms。 --- ## 着色器调度模式 每次矩阵乘法是一个独立的计算调度。以 TinyLlama（22 层）单个解码步骤为例： ``` 每 token 的大致调度次数： 22 × (Q投影 + K投影 + V投影 + 注意力输出 + FFN_gate/up + FFN_down) ≈ 132 次调度 + 22 次 RMSNorm + 22 次 RoPE + 22 次注意力得分 + 22 次 Softmax ≈ 共 220 次 GPU 调度/token ``` 每次调度使用 `(16, 16, 1)` 的工作组，最大计算调用数为 256/工作组。命令编码器将所有调度打包为**每 token 一次**命令缓冲区提交——而不是每次调度单独提交。这对性能至关重要：每 token 一次 GPU 往返，而非 220 次。 **调试调度**：设置 `SHIMMY_DEBUG_RAW=1` 可记录管线各点的原始激活值（极度冗长——仅用于着色器调试）。 --- ## 输出头 最终层产生形状为 `[1, n_embd]` 的激活值（最后位置）。将其与**输出嵌入权重**（`output.weight`，形状 `[vocab_size, n_embd]`）相乘，得到 logits。 输出权重通常是显存占用最大的张量：以词表 32K、维度 4096 的模型为例，`output.weight` 全精度下为 `32768 × 4096 × 4 ≈ 512 MB`。GGUF 的 Q6_K 格式下约 210 MB。 **WebGPU 2 GB 缓冲区限制**：输出权重必须放入单个 GPU 缓冲区。词表规模超大的模型（如 Gemma-2 词表 256K）的输出嵌入超过 2 GB，无法加载。这是已知限制——参见[故障排查指南](TROUBLESHOOTING.md)。 --- ## 性能参考 **Token 生成速度**主要取决于： 1. GPU 显存带宽（不是算力 FLOPS）——权重读取是主要开销 2. 模型大小——参数越多，每 token 读取的字节越多 3. 上下文长度——解码阶段注意力得分按 O(n) 规模增长 **RTX 3060 12GB 实测基准：** | 模型 | 上下文 | Token/秒 | |------|--------|---------| | TinyLlama 1.1B Q4_0 | 2048 | ~35-50 | | Llama-3.2-1B Q4_K_M | 2048 | ~30-45 | | Llama-3.2-3B Q4_K_M | 2048 | ~12-18 | | Phi-2 2.7B Q4_K_M | 2048 | ~10-15 | 预填充通常比解码快 3-10 倍（每 token），因为注意力和 FFN 可以在序列维度上并行处理多个 token。 --- ## 延伸阅读 - [量化格式详解](QUANTIZATION.md) — 着色器内部反量化 - [扩展上下文窗口](EXTENDED_CONTEXT.md) — YaRN RoPE 缩放实现 - [故障排查指南](TROUBLESHOOTING.md) — GPU 故障调试 --- > 💝 **如果 Shimmy 对您有帮助，欢迎[赞助支持](https://github.com/sponsors/Michael-A-Kuykendall)——所有款项 100% 用于保持项目永久免费。** --- METRICS --- - Files Extracted: 11 - Estimated Token Budget: ~20320 tokens - Recency Window: Active (< 180 days) - Canonical Reference: https://codewiki.google/github.com/Michael-A-Kuykendall/shimmy