{"owner":"EricLBuehler","repo":"mistral.rs","hasSkills":true,"hasMcp":false,"mcpConfig":null,"found":["AGENTS.md","CLAUDE.md"],"skills":{"AGENTS.md":"<!-- AGENTS.md: Guidance for AI agents to navigate, build, test, and contribute to this repository -->\n# AGENTS\n\nThis file provides instructions for AI agents to understand the layout of the `mistral.rs` repository, run builds/tests, and follow project conventions.\n\n## Repository Structure\n\n- `/mistralrs/`           : Main Rust crate (text & multimodal inference API)\n- `/mistralrs-core/`      : Core inference logic and tensor operations (text models)\n- `/mistralrs-vision/`    : Image processing utilities (resizing, preprocessing for multimodal models)\n- `/mistralrs-quant/`     : Quantization support (ISQ, GGUF, GPTQ, AWQ, FP8, HQQ, etc.)\n- `/mistralrs-paged-attn/`: PagedAttention implementation\n- `/mistralrs-pyo3/`      : Python bindings (PyO3)\n- `/mistralrs-cli/`       : Unified CLI binary (commands: run, serve, bench, from-config)\n- `/mistralrs-server-core/`: Shared server core logic\n- `/docs/`             : Astro/Starlight documentation site (deployed to GitHub Pages)\n- `/examples/`            : Usage examples (Rust, Python, server samples, notebooks)\n- `/chat_templates/`      : Chat formatting templates (JSON/Jinja)\n- `/scripts/`             : Utility scripts (e.g., AWQ conversion)\n  \n## Feature Organization\n\nMistral.rs supports multiple model types and advanced features via dedicated crates and CLI subcommands:\n\n- **Text Inference**\n  - Crate: `mistralrs-core` (low-level ops), `mistralrs` (API wrapper)\n  - CLI: `mistralrs run -m <model>` or `mistralrs serve -m <model>` (auto-detects model type)\n  - Docs: `docs/src/content/docs/guides/customize/sampling.md`, `docs/src/content/docs/guides/agents/`\n- **Multimodal Models**\n  - Crate: `mistralrs-vision`\n  - CLI: `mistralrs run -m <model>` (auto-detects multimodal models)\n  - Docs: `docs/src/content/docs/explanation/multimodal-pipeline.md`, `docs/src/content/docs/reference/supported-models.md`\n- **Diffusion Models**\n  - CLI: `mistralrs run -m <model>` (auto-detects diffusion models)\n  - Docs: `docs/src/content/docs/reference/supported-models.md`\n- **Speech Models**\n  - CLI: `mistralrs run -m <model>` (auto-detects speech models)\n  - Docs: `docs/src/content/docs/reference/supported-models.md`\n- **Quantization & ISQ**\n  - Crate: `mistralrs-quant`\n  - Docs: `docs/src/content/docs/reference/quantization-types.md`, `docs/src/content/docs/explanation/quantization-tradeoffs.md`\n  - Conversion Script: `scripts/convert_awq_marlin.py`\n- **Paged Attention**\n  - Crate: `mistralrs-paged-attn`\n  - Docs: `docs/src/content/docs/explanation/paged-attention.md`, `docs/src/content/docs/guides/perf/use-paged-attention.md`\n- **Adapters & LoRA/X-LoRA**\n  - Docs: `docs/src/content/docs/guides/customize/lora-adapters.md`\n- **Mixture of Experts (AnyMoE)**\n  - Docs: `docs/src/content/docs/guides/customize/anymoe.md`\n\n## Building\n\n1. Install Rust via rustup (Rust 2021 edition).\n2. Choose optional features (e.g., `cuda`, `flash-attn`, `cudnn`, `metal`, `mkl`, `accelerate`).\n3. Build the entire workspace:\n   ```bash\n   cargo build --workspace --release --features \"<features>\"\n   ```\n4. Or build/install only the CLI binary:\n   ```bash\n   cargo build --release --package mistralrs-cli --features \"<features>\"\n   cargo install --path mistralrs-cli --features \"<features>\"\n   ```\n\n## Models\n\nWhen integrating a new model, make sure it respects all of the varbuilder `.pp` calls. In Candle, a VarBuilder maintains an internal path vector that acts like a “current working directory” for model weights; every call to pp(\"sub\") (alias for push_prefix) clones the builder and appends sub, so successive calls accumulate a dotted prefix such as transformer.h.0 while leaving the original builder untouched . When you eventually call get(...), Candle joins that prefix with the tensor name (prefix + \".\" + name) and looks it up in the checkpoint backend, producing keys that exactly match the dot-separated names emitted by PyTorch’s state_dict/named_parameters, which means PyTorch-trained weights can be loaded without any renaming  . This lets you recreate the PyTorch module tree in Rust by “walking” it: e.g. vb.pp(\"word_embeddings\") grabs word_embeddings.*, while a chain like vb.pp(\"encoder\").pp(\"layers\").pp(i.to_string()) targets keys such as encoder.layers.0.*, exactly as shown in community tutorials porting Transformers models to Candle  . As one maintainer put it, the prefix system lets you “cd” around the parameter hierarchy, giving a lightweight namespace mechanism that keeps Candle fully compatible with PyTorch naming conventions while remaining ergonomic to use.\n\nYou should also look for a model.safetensors.index.json file for the model at hand to verify correct structure.\n\n## Testing\n\n- Core test suite (requires HF token for some tests):\n  ```bash\n  export HF_TOKEN=<your_token>  # or TESTS_HF_TOKEN for CI parity\n  cargo test -p mistralrs-core -p mistralrs-quant -p mistralrs-vision\n  ```\n- Run all tests across workspace (may skip some crates without tests):\n  ```bash\n  cargo test --workspace\n  ```\n\nYou should *always* run `cargo check`/`cargo c` before returning to make sure code compiles. If code does not compile, only make edits.\n\nAvoid returning TODOs.\n\n## Formatting & Linting\n\n- Format all Rust code:\n  ```bash\n  cargo fmt --all\n  make fmt       # also formats Python/CUDA/C++ files via ruff, clang-format\n  ```\n- Lint with Clippy:\n  ```bash\n  cargo clippy --workspace --tests --examples -- -D warnings\n  ```\n\n## Documentation\n\n- Generate Rust docs for all crates:\n  ```bash\n  cargo doc --workspace\n  ```\n- Preview Rust API docs at `target/doc/`.\n- Refer to `/docs/src/content/docs/` for in-depth guides. The site builds with `cd docs && npm run build` and deploys to GitHub Pages via `.github/workflows/docs.yml`.\n\n## Examples\n\n- Rust examples: `mistralrs/examples/`\n- Python examples: `examples/python/`\n- Server samples: `examples/server/`\n- Run Python scripts:\n  ```bash\n  python3 examples/python/<script>.py\n  ```\n- Run CLI:\n  ```bash\n  mistralrs run -m <model>        # Interactive mode\n  mistralrs serve -p 1234 -m <model>  # Server mode\n  mistralrs bench -m <model>      # Benchmarking\n  ```\n\n## CI Parity\n\nThe CI pipeline is defined in `.github/workflows/ci.yml` and includes:\n  - `cargo check` for all targets\n  - `cargo test` on core crates\n  - `cargo fmt -- --check`\n  - `cargo clippy -D warnings`\n  - `cargo doc`\n  - Typos check (`crate-ci/typos`)\n\n## Contribution Conventions\n\n- Follow Rust 2021 idioms, keep code minimal and focused.\n- Update `/docs/src/content/docs/` and examples when adding features or breaking changes.\n- Add tests and examples for new functionality.\n- Commit messages should be clear and follow conventional style where possible.\n  ```\n  feat(crate): describe new feature\n  fix(crate): describe bug fix\n  docs: update docs for ...\n  ```\n\n### Code Style (Extremely important & convention for this codebase)\n\n**Comments.** Default to none. Only add when the *why* isn't obvious from the code: hidden constraints, invariants, surprising edge cases, references to a spec/HF source. Never paraphrase what the next line does, never restate the function name, never narrate steps.\n\n- Multi-line comments are discouraged in code, and only really allowed in documentation or where they are the best way to communicate information.\n- Code comments should be one line each, up to ~120 cols. No multi-paragraph `///` blocks, no bulleted lists in doc comments, no `// === Section ===` or `// ── Section ──` banners.\n- Tone for inline code comments should be terse, casual, and never explaining what the code directly below does.\n- Only include code comments if they add new information, and never just for the sake of it.\n\n- Unless otherwise instructed, use ASCII only. No em-dashes (`—`), en-dashes (`–`), ellipses (`…`), smart quotes, or box-drawing characters. Do not use `--`. It's ok to use `...`, `\"`, `'` when appropriate.\n- Don't reference the current task / PR / fix / commit in comments — that belongs in the PR description and rots as the codebase evolves.\n- Trailing inline annotations like `// already sent above` are fine when terse.\n\n**Magic values.** Hoist durations, sizes, sentinels, and other constants to named `const`s at the top of the file. A sentinel value that crosses module boundaries (e.g. one place sets `Some(0)`, another checks for it) must be a `pub const`, not a literal both sides happen to share.\n\n**Function shape.** When a function passes 6+ args, prefer wrapping the invariants in a small context struct (e.g. `DispatchCtx<'a>`). Don't add error handling, fallbacks, or validation for scenarios that can't actually occur — trust internal code and framework guarantees. Don't add backwards-compatibility shims unless explicitly asked.\n\n---\n*This AGENTS.md file is intended solely to improve AI-driven assistance and does not affect runtime behavior.*\n","CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nmistral.rs is a blazing-fast LLM inference engine written in Rust. It supports text, multimodal, image generation, and speech models with Rust and Python SDKs, plus OpenAI HTTP and MCP APIs.\n\n## Essential Commands\n\n### Building\n```bash\n# Basic release build\ncargo build --release\n\n# With CUDA support (Linux)\ncargo build --release --features \"cuda flash-attn cudnn\"\n\n# With Metal support (macOS)\ncargo build --release --features metal\n\n# Install CLI binary\ncargo install --path mistralrs-cli --features <features>\n```\n\n### Testing & Quality\n```bash\n# Run core tests\ncargo test -p mistralrs-core -p mistralrs-quant -p mistralrs-vision\n\n# Format code (uses rustfmt, ruff, clang-format)\nmake fmt\n\n# Check formatting\ncargo fmt --all -- --check\n\n# Run clippy\ncargo clippy --workspace --tests --examples -- -D warnings\n```\n\n### Running Models\n```bash\n# Run interactive mode (model type auto-detected)\nmistralrs run -m <model_id>\n\n# Run with GGUF quantized model\nmistralrs run --format gguf -m <repo> -f <file>\n\n# Run server\nmistralrs serve -p 1234 -m <model_id>\n\n# Run server (built-in web UI is on by default at /ui; pass --no-ui to disable)\nmistralrs serve -m <model_id>\n\n# Run benchmarks\nmistralrs bench -m <model_id>\n```\n\n## Models\n\nWhen integrating a new model, make sure it respects all of the varbuilder `.pp` calls. In Candle, a VarBuilder maintains an internal path vector that acts like a “current working directory” for model weights; every call to pp(\"sub\") (alias for push_prefix) clones the builder and appends sub, so successive calls accumulate a dotted prefix such as transformer.h.0 while leaving the original builder untouched . When you eventually call get(...), Candle joins that prefix with the tensor name (prefix + \".\" + name) and looks it up in the checkpoint backend, producing keys that exactly match the dot-separated names emitted by PyTorch’s state_dict/named_parameters, which means PyTorch-trained weights can be loaded without any renaming  . This lets you recreate the PyTorch module tree in Rust by “walking” it: e.g. vb.pp(\"word_embeddings\") grabs word_embeddings.*, while a chain like vb.pp(\"encoder\").pp(\"layers\").pp(i.to_string()) targets keys such as encoder.layers.0.*, exactly as shown in community tutorials porting Transformers models to Candle  . As one maintainer put it, the prefix system lets you “cd” around the parameter hierarchy, giving a lightweight namespace mechanism that keeps Candle fully compatible with PyTorch naming conventions while remaining ergonomic to use.\n\nYou should also look for a model.safetensors.index.json file for the model at hand to verify correct structure.\n\n## Architecture Overview\n\n### Workspace Structure\n- `mistralrs-core/` - Core inference engine, model implementations, pipelines\n- `mistralrs-cli/` - Unified CLI binary (commands: run, serve, bench, from-config)\n- `mistralrs-server-core/` - HTTP server routing, OpenAI API implementation\n- `mistralrs-pyo3/` - Python SDK (PyO3 bindings)\n- `mistralrs/` - Rust SDK (high-level crate)\n- `mistralrs-vision/` - Image processing utilities\n- `mistralrs-quant/` - Quantization implementations (ISQ, GGUF, GPTQ, etc.)\n- `mistralrs-paged-attn/` - PagedAttention implementation\n- `mistralrs-audio/` - Audio processing\n- `mistralrs-mcp/` - Model Context Protocol client\n\n### Key Design Patterns\n\n1. **Pipeline Architecture**: All models implement the `Pipeline` trait in `mistralrs-core/src/pipeline/mod.rs`. Different model types (Plain, GGUF, GGML, Multimodal) have their own pipeline implementations.\n\n2. **Model Loading**: Models are loaded through `Loader` traits that handle different formats and quantizations. See `mistralrs-core/src/loader.rs`.\n\n3. **Request Handling**: The server uses message passing with `MistralRs` struct managing a background thread pool. Requests flow through `mistralrs-core/src/engine/mod.rs`.\n\n4. **Device Management**: Automatic and manual device mapping for multi-GPU setups handled in `mistralrs-core/src/device_map.rs`.\n\n### Adding New Features\n\nWhen adding new model architectures:\n1. Implement the model in `mistralrs-core/src/models/`\n2. Add pipeline support in `mistralrs-core/src/pipeline/`\n3. Update model detection in `mistralrs-core/src/pipeline/normal.rs`\n4. Add architecture enum variant in `mistralrs-core/src/lib.rs`\n5. Update CLI args in `mistralrs-cli/src/main.rs`\n\nWhen adding new quantization methods:\n1. Implement in `mistralrs-quant/src/`\n2. Add to quantization loading logic in pipelines\n3. Update documentation in `docs/src/content/docs/reference/quantization-types.md`\n\n### Important Files to Know\n\n- `mistralrs-core/src/engine/mod.rs` - Main engine orchestration\n- `mistralrs-core/src/pipeline/mod.rs` - Pipeline trait and common logic\n- `mistralrs-server-core/src/routes.rs` - HTTP API endpoints\n- `mistralrs-pyo3/src/lib.rs` - Python SDK entry point\n- `mistralrs/examples/` - Usage examples for Rust SDK\n\n### Pull Requests\n\nNever include a \"Test plan\" section in PR descriptions.\n\n### Code Style (Extremely important & convention for this codebase)\n\n**Comments.** Default to none. Only add when the *why* isn't obvious from the code: hidden constraints, invariants, surprising edge cases, references to a spec/HF source. Never paraphrase what the next line does, never restate the function name, never narrate steps.\n\n- Multi-line comments are discouraged in code, and only really allowed in documentation or where they are the best way to communicate information.\n- Code comments should be one line each, up to ~120 cols. No multi-paragraph `///` blocks, no bulleted lists in doc comments, no `// === Section ===` or `// ── Section ──` banners.\n- Tone for inline code comments should be terse, casual, and never explaining what the code directly below does.\n- Only include code comments if they add new information, and never just for the sake of it.\n\n- Unless otherwise instructed, use ASCII only. No em-dashes (`—`), en-dashes (`–`), ellipses (`…`), smart quotes, or box-drawing characters. Do not use `--`. It's ok to use `...`, `\"`, `'` when appropriate.\n- Don't reference the current task / PR / fix / commit in comments — that belongs in the PR description and rots as the codebase evolves.\n- Trailing inline annotations like `// already sent above` are fine when terse.\n\n**Magic values.** Hoist durations, sizes, sentinels, and other constants to named `const`s at the top of the file. A sentinel value that crosses module boundaries (e.g. one place sets `Some(0)`, another checks for it) must be a `pub const`, not a literal both sides happen to share.\n\n**Function shape.** When a function passes 6+ args, prefer wrapping the invariants in a small context struct (e.g. `DispatchCtx<'a>`). Don't add error handling, fallbacks, or validation for scenarios that can't actually occur — trust internal code and framework guarantees. Don't add backwards-compatibility shims unless explicitly asked.\n\n### Testing Approach\n\nYou should *always* run `cargo check`/`cargo c` before returning to make sure code compiles. If code does not compile, only make edits.\n\nAvoid returning TODOs.\n\n- Unit tests are colocated with source files\n- Integration tests in `tests/` directories\n- Use `cargo test -p <crate>` to test specific components\n- Python tests require building and installing the package first\n\n### Common Pitfalls\n\n1. **Feature Flags**: Many features are gated behind Cargo features. Always check what features are needed for your use case.\n2. **Device Indices**: CUDA device selection uses 0-based indexing\n3. **Chat Templates**: Models may need specific chat templates - check `chat_templates/` directory\n4. **Quantization**: Different quantization methods have different hardware requirements\n5. **Never use `Tensor::{from_vec,arange}` in hot loops**: `Tensor::{from_vec,arange}` with a GPU device causes a CPU-to-GPU sync. If you need a small tensor on GPU during forward, either precompute it at model init or start of forward pass.\n\n### Vision/Audio Model Pitfalls\n\n6. **Vision encoder attention must be bidirectional (non-causal)**:  `Sdpa.run_attention` with `flash_params: None` defaults to `causal = seq_len > 1` on the CUDA flash-attn path, which silently breaks vision/audio encoders. Always pass `FlashParams { causal: false, cumulative_seqlens_q: HashMap::new(), cumulative_seqlens_k: HashMap::new(), max_q: 0, max_k: 0 }` with `Some(&flash_params)` for any encoder that needs bidirectional attention. The empty `cumulative_seqlens` cause the flash backend to use the non-varlen kernel path, avoiding any tensor allocation in the forward pass.\n\n7. **`torch.bucketize(right=True)` requires `Ok(i) => i + 1`**: Rust's `binary_search_by` returns `Ok(i)` at the found position (bisect_left semantics). For `right=True` (bisect_right), you must use `Ok(i) => i + 1` to insert after equal elements. `Err(i) => i` is correct for both.\n\n8. **Mistral `consolidated.safetensors` stores Q/K weights with interleaved head dimensions**: When loading from Mistral-native `consolidated.safetensors` (as opposed to HF-converted `model.safetensors`), the Q and K projection weights use an interleaved layout within each head: `[x0, x_{d/2}, x1, x_{d/2+1}, ...]` instead of the sequential HF layout `[x0, x1, ..., x_{d/2-1}, x_{d/2}, ...]`. This means you must use `is_gptx=false` (GPT-J/adjacent-pair style) for `RotaryEmbedding`, NOT `is_gptx=true` (GPT-NeoX/half-split style). Using the wrong RoPE style produces completely wrong attention outputs (cosine similarity ~0.02 with reference). To diagnose: compare a Q or K weight tensor between `consolidated.safetensors` and `model.safetensors` — if they differ (cosine ~0.02), apply the un-interleave: `reshape(n_heads, head_dim/2, 2, dim).permute(0,2,1,3)` and verify cosine ~1.0.\n\n9. **Causal Conv1d padding formula**: For causal convolution (left-pad only, no right-pad), the correct left padding is `effective_kernel_size - stride`, NOT `(kernel_size - 1) * dilation` (which is the total padding for non-causal). For example, with kernel_size=3, stride=2, dilation=1: left_pad = 3 - 2 = 1, not 2. Verify against the HF model's `VoxtralRealtimeCausalConv1d` or equivalent source.\n"},"files":{"AGENTS.md":"<!-- AGENTS.md: Guidance for AI agents to navigate, build, test, and contribute to this repository -->\n# AGENTS\n\nThis file provides instructions for AI agents to understand the layout of the `mistral.rs` repository, run builds/tests, and follow project conventions.\n\n## Repository Structure\n\n- `/mistralrs/`           : Main Rust crate (text & multimodal inference API)\n- `/mistralrs-core/`      : Core inference logic and tensor operations (text models)\n- `/mistralrs-vision/`    : Image processing utilities (resizing, preprocessing for multimodal models)\n- `/mistralrs-quant/`     : Quantization support (ISQ, GGUF, GPTQ, AWQ, FP8, HQQ, etc.)\n- `/mistralrs-paged-attn/`: PagedAttention implementation\n- `/mistralrs-pyo3/`      : Python bindings (PyO3)\n- `/mistralrs-cli/`       : Unified CLI binary (commands: run, serve, bench, from-config)\n- `/mistralrs-server-core/`: Shared server core logic\n- `/docs/`             : Astro/Starlight documentation site (deployed to GitHub Pages)\n- `/examples/`            : Usage examples (Rust, Python, server samples, notebooks)\n- `/chat_templates/`      : Chat formatting templates (JSON/Jinja)\n- `/scripts/`             : Utility scripts (e.g., AWQ conversion)\n  \n## Feature Organization\n\nMistral.rs supports multiple model types and advanced features via dedicated crates and CLI subcommands:\n\n- **Text Inference**\n  - Crate: `mistralrs-core` (low-level ops), `mistralrs` (API wrapper)\n  - CLI: `mistralrs run -m <model>` or `mistralrs serve -m <model>` (auto-detects model type)\n  - Docs: `docs/src/content/docs/guides/customize/sampling.md`, `docs/src/content/docs/guides/agents/`\n- **Multimodal Models**\n  - Crate: `mistralrs-vision`\n  - CLI: `mistralrs run -m <model>` (auto-detects multimodal models)\n  - Docs: `docs/src/content/docs/explanation/multimodal-pipeline.md`, `docs/src/content/docs/reference/supported-models.md`\n- **Diffusion Models**\n  - CLI: `mistralrs run -m <model>` (auto-detects diffusion models)\n  - Docs: `docs/src/content/docs/reference/supported-models.md`\n- **Speech Models**\n  - CLI: `mistralrs run -m <model>` (auto-detects speech models)\n  - Docs: `docs/src/content/docs/reference/supported-models.md`\n- **Quantization & ISQ**\n  - Crate: `mistralrs-quant`\n  - Docs: `docs/src/content/docs/reference/quantization-types.md`, `docs/src/content/docs/explanation/quantization-tradeoffs.md`\n  - Conversion Script: `scripts/convert_awq_marlin.py`\n- **Paged Attention**\n  - Crate: `mistralrs-paged-attn`\n  - Docs: `docs/src/content/docs/explanation/paged-attention.md`, `docs/src/content/docs/guides/perf/use-paged-attention.md`\n- **Adapters & LoRA/X-LoRA**\n  - Docs: `docs/src/content/docs/guides/customize/lora-adapters.md`\n- **Mixture of Experts (AnyMoE)**\n  - Docs: `docs/src/content/docs/guides/customize/anymoe.md`\n\n## Building\n\n1. Install Rust via rustup (Rust 2021 edition).\n2. Choose optional features (e.g., `cuda`, `flash-attn`, `cudnn`, `metal`, `mkl`, `accelerate`).\n3. Build the entire workspace:\n   ```bash\n   cargo build --workspace --release --features \"<features>\"\n   ```\n4. Or build/install only the CLI binary:\n   ```bash\n   cargo build --release --package mistralrs-cli --features \"<features>\"\n   cargo install --path mistralrs-cli --features \"<features>\"\n   ```\n\n## Models\n\nWhen integrating a new model, make sure it respects all of the varbuilder `.pp` calls. In Candle, a VarBuilder maintains an internal path vector that acts like a “current working directory” for model weights; every call to pp(\"sub\") (alias for push_prefix) clones the builder and appends sub, so successive calls accumulate a dotted prefix such as transformer.h.0 while leaving the original builder untouched . When you eventually call get(...), Candle joins that prefix with the tensor name (prefix + \".\" + name) and looks it up in the checkpoint backend, producing keys that exactly match the dot-separated names emitted by PyTorch’s state_dict/named_parameters, which means PyTorch-trained weights can be loaded without any renaming  . This lets you recreate the PyTorch module tree in Rust by “walking” it: e.g. vb.pp(\"word_embeddings\") grabs word_embeddings.*, while a chain like vb.pp(\"encoder\").pp(\"layers\").pp(i.to_string()) targets keys such as encoder.layers.0.*, exactly as shown in community tutorials porting Transformers models to Candle  . As one maintainer put it, the prefix system lets you “cd” around the parameter hierarchy, giving a lightweight namespace mechanism that keeps Candle fully compatible with PyTorch naming conventions while remaining ergonomic to use.\n\nYou should also look for a model.safetensors.index.json file for the model at hand to verify correct structure.\n\n## Testing\n\n- Core test suite (requires HF token for some tests):\n  ```bash\n  export HF_TOKEN=<your_token>  # or TESTS_HF_TOKEN for CI parity\n  cargo test -p mistralrs-core -p mistralrs-quant -p mistralrs-vision\n  ```\n- Run all tests across workspace (may skip some crates without tests):\n  ```bash\n  cargo test --workspace\n  ```\n\nYou should *always* run `cargo check`/`cargo c` before returning to make sure code compiles. If code does not compile, only make edits.\n\nAvoid returning TODOs.\n\n## Formatting & Linting\n\n- Format all Rust code:\n  ```bash\n  cargo fmt --all\n  make fmt       # also formats Python/CUDA/C++ files via ruff, clang-format\n  ```\n- Lint with Clippy:\n  ```bash\n  cargo clippy --workspace --tests --examples -- -D warnings\n  ```\n\n## Documentation\n\n- Generate Rust docs for all crates:\n  ```bash\n  cargo doc --workspace\n  ```\n- Preview Rust API docs at `target/doc/`.\n- Refer to `/docs/src/content/docs/` for in-depth guides. The site builds with `cd docs && npm run build` and deploys to GitHub Pages via `.github/workflows/docs.yml`.\n\n## Examples\n\n- Rust examples: `mistralrs/examples/`\n- Python examples: `examples/python/`\n- Server samples: `examples/server/`\n- Run Python scripts:\n  ```bash\n  python3 examples/python/<script>.py\n  ```\n- Run CLI:\n  ```bash\n  mistralrs run -m <model>        # Interactive mode\n  mistralrs serve -p 1234 -m <model>  # Server mode\n  mistralrs bench -m <model>      # Benchmarking\n  ```\n\n## CI Parity\n\nThe CI pipeline is defined in `.github/workflows/ci.yml` and includes:\n  - `cargo check` for all targets\n  - `cargo test` on core crates\n  - `cargo fmt -- --check`\n  - `cargo clippy -D warnings`\n  - `cargo doc`\n  - Typos check (`crate-ci/typos`)\n\n## Contribution Conventions\n\n- Follow Rust 2021 idioms, keep code minimal and focused.\n- Update `/docs/src/content/docs/` and examples when adding features or breaking changes.\n- Add tests and examples for new functionality.\n- Commit messages should be clear and follow conventional style where possible.\n  ```\n  feat(crate): describe new feature\n  fix(crate): describe bug fix\n  docs: update docs for ...\n  ```\n\n### Code Style (Extremely important & convention for this codebase)\n\n**Comments.** Default to none. Only add when the *why* isn't obvious from the code: hidden constraints, invariants, surprising edge cases, references to a spec/HF source. Never paraphrase what the next line does, never restate the function name, never narrate steps.\n\n- Multi-line comments are discouraged in code, and only really allowed in documentation or where they are the best way to communicate information.\n- Code comments should be one line each, up to ~120 cols. No multi-paragraph `///` blocks, no bulleted lists in doc comments, no `// === Section ===` or `// ── Section ──` banners.\n- Tone for inline code comments should be terse, casual, and never explaining what the code directly below does.\n- Only include code comments if they add new information, and never just for the sake of it.\n\n- Unless otherwise instructed, use ASCII only. No em-dashes (`—`), en-dashes (`–`), ellipses (`…`), smart quotes, or box-drawing characters. Do not use `--`. It's ok to use `...`, `\"`, `'` when appropriate.\n- Don't reference the current task / PR / fix / commit in comments — that belongs in the PR description and rots as the codebase evolves.\n- Trailing inline annotations like `// already sent above` are fine when terse.\n\n**Magic values.** Hoist durations, sizes, sentinels, and other constants to named `const`s at the top of the file. A sentinel value that crosses module boundaries (e.g. one place sets `Some(0)`, another checks for it) must be a `pub const`, not a literal both sides happen to share.\n\n**Function shape.** When a function passes 6+ args, prefer wrapping the invariants in a small context struct (e.g. `DispatchCtx<'a>`). Don't add error handling, fallbacks, or validation for scenarios that can't actually occur — trust internal code and framework guarantees. Don't add backwards-compatibility shims unless explicitly asked.\n\n---\n*This AGENTS.md file is intended solely to improve AI-driven assistance and does not affect runtime behavior.*\n","CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nmistral.rs is a blazing-fast LLM inference engine written in Rust. It supports text, multimodal, image generation, and speech models with Rust and Python SDKs, plus OpenAI HTTP and MCP APIs.\n\n## Essential Commands\n\n### Building\n```bash\n# Basic release build\ncargo build --release\n\n# With CUDA support (Linux)\ncargo build --release --features \"cuda flash-attn cudnn\"\n\n# With Metal support (macOS)\ncargo build --release --features metal\n\n# Install CLI binary\ncargo install --path mistralrs-cli --features <features>\n```\n\n### Testing & Quality\n```bash\n# Run core tests\ncargo test -p mistralrs-core -p mistralrs-quant -p mistralrs-vision\n\n# Format code (uses rustfmt, ruff, clang-format)\nmake fmt\n\n# Check formatting\ncargo fmt --all -- --check\n\n# Run clippy\ncargo clippy --workspace --tests --examples -- -D warnings\n```\n\n### Running Models\n```bash\n# Run interactive mode (model type auto-detected)\nmistralrs run -m <model_id>\n\n# Run with GGUF quantized model\nmistralrs run --format gguf -m <repo> -f <file>\n\n# Run server\nmistralrs serve -p 1234 -m <model_id>\n\n# Run server (built-in web UI is on by default at /ui; pass --no-ui to disable)\nmistralrs serve -m <model_id>\n\n# Run benchmarks\nmistralrs bench -m <model_id>\n```\n\n## Models\n\nWhen integrating a new model, make sure it respects all of the varbuilder `.pp` calls. In Candle, a VarBuilder maintains an internal path vector that acts like a “current working directory” for model weights; every call to pp(\"sub\") (alias for push_prefix) clones the builder and appends sub, so successive calls accumulate a dotted prefix such as transformer.h.0 while leaving the original builder untouched . When you eventually call get(...), Candle joins that prefix with the tensor name (prefix + \".\" + name) and looks it up in the checkpoint backend, producing keys that exactly match the dot-separated names emitted by PyTorch’s state_dict/named_parameters, which means PyTorch-trained weights can be loaded without any renaming  . This lets you recreate the PyTorch module tree in Rust by “walking” it: e.g. vb.pp(\"word_embeddings\") grabs word_embeddings.*, while a chain like vb.pp(\"encoder\").pp(\"layers\").pp(i.to_string()) targets keys such as encoder.layers.0.*, exactly as shown in community tutorials porting Transformers models to Candle  . As one maintainer put it, the prefix system lets you “cd” around the parameter hierarchy, giving a lightweight namespace mechanism that keeps Candle fully compatible with PyTorch naming conventions while remaining ergonomic to use.\n\nYou should also look for a model.safetensors.index.json file for the model at hand to verify correct structure.\n\n## Architecture Overview\n\n### Workspace Structure\n- `mistralrs-core/` - Core inference engine, model implementations, pipelines\n- `mistralrs-cli/` - Unified CLI binary (commands: run, serve, bench, from-config)\n- `mistralrs-server-core/` - HTTP server routing, OpenAI API implementation\n- `mistralrs-pyo3/` - Python SDK (PyO3 bindings)\n- `mistralrs/` - Rust SDK (high-level crate)\n- `mistralrs-vision/` - Image processing utilities\n- `mistralrs-quant/` - Quantization implementations (ISQ, GGUF, GPTQ, etc.)\n- `mistralrs-paged-attn/` - PagedAttention implementation\n- `mistralrs-audio/` - Audio processing\n- `mistralrs-mcp/` - Model Context Protocol client\n\n### Key Design Patterns\n\n1. **Pipeline Architecture**: All models implement the `Pipeline` trait in `mistralrs-core/src/pipeline/mod.rs`. Different model types (Plain, GGUF, GGML, Multimodal) have their own pipeline implementations.\n\n2. **Model Loading**: Models are loaded through `Loader` traits that handle different formats and quantizations. See `mistralrs-core/src/loader.rs`.\n\n3. **Request Handling**: The server uses message passing with `MistralRs` struct managing a background thread pool. Requests flow through `mistralrs-core/src/engine/mod.rs`.\n\n4. **Device Management**: Automatic and manual device mapping for multi-GPU setups handled in `mistralrs-core/src/device_map.rs`.\n\n### Adding New Features\n\nWhen adding new model architectures:\n1. Implement the model in `mistralrs-core/src/models/`\n2. Add pipeline support in `mistralrs-core/src/pipeline/`\n3. Update model detection in `mistralrs-core/src/pipeline/normal.rs`\n4. Add architecture enum variant in `mistralrs-core/src/lib.rs`\n5. Update CLI args in `mistralrs-cli/src/main.rs`\n\nWhen adding new quantization methods:\n1. Implement in `mistralrs-quant/src/`\n2. Add to quantization loading logic in pipelines\n3. Update documentation in `docs/src/content/docs/reference/quantization-types.md`\n\n### Important Files to Know\n\n- `mistralrs-core/src/engine/mod.rs` - Main engine orchestration\n- `mistralrs-core/src/pipeline/mod.rs` - Pipeline trait and common logic\n- `mistralrs-server-core/src/routes.rs` - HTTP API endpoints\n- `mistralrs-pyo3/src/lib.rs` - Python SDK entry point\n- `mistralrs/examples/` - Usage examples for Rust SDK\n\n### Pull Requests\n\nNever include a \"Test plan\" section in PR descriptions.\n\n### Code Style (Extremely important & convention for this codebase)\n\n**Comments.** Default to none. Only add when the *why* isn't obvious from the code: hidden constraints, invariants, surprising edge cases, references to a spec/HF source. Never paraphrase what the next line does, never restate the function name, never narrate steps.\n\n- Multi-line comments are discouraged in code, and only really allowed in documentation or where they are the best way to communicate information.\n- Code comments should be one line each, up to ~120 cols. No multi-paragraph `///` blocks, no bulleted lists in doc comments, no `// === Section ===` or `// ── Section ──` banners.\n- Tone for inline code comments should be terse, casual, and never explaining what the code directly below does.\n- Only include code comments if they add new information, and never just for the sake of it.\n\n- Unless otherwise instructed, use ASCII only. No em-dashes (`—`), en-dashes (`–`), ellipses (`…`), smart quotes, or box-drawing characters. Do not use `--`. It's ok to use `...`, `\"`, `'` when appropriate.\n- Don't reference the current task / PR / fix / commit in comments — that belongs in the PR description and rots as the codebase evolves.\n- Trailing inline annotations like `// already sent above` are fine when terse.\n\n**Magic values.** Hoist durations, sizes, sentinels, and other constants to named `const`s at the top of the file. A sentinel value that crosses module boundaries (e.g. one place sets `Some(0)`, another checks for it) must be a `pub const`, not a literal both sides happen to share.\n\n**Function shape.** When a function passes 6+ args, prefer wrapping the invariants in a small context struct (e.g. `DispatchCtx<'a>`). Don't add error handling, fallbacks, or validation for scenarios that can't actually occur — trust internal code and framework guarantees. Don't add backwards-compatibility shims unless explicitly asked.\n\n### Testing Approach\n\nYou should *always* run `cargo check`/`cargo c` before returning to make sure code compiles. If code does not compile, only make edits.\n\nAvoid returning TODOs.\n\n- Unit tests are colocated with source files\n- Integration tests in `tests/` directories\n- Use `cargo test -p <crate>` to test specific components\n- Python tests require building and installing the package first\n\n### Common Pitfalls\n\n1. **Feature Flags**: Many features are gated behind Cargo features. Always check what features are needed for your use case.\n2. **Device Indices**: CUDA device selection uses 0-based indexing\n3. **Chat Templates**: Models may need specific chat templates - check `chat_templates/` directory\n4. **Quantization**: Different quantization methods have different hardware requirements\n5. **Never use `Tensor::{from_vec,arange}` in hot loops**: `Tensor::{from_vec,arange}` with a GPU device causes a CPU-to-GPU sync. If you need a small tensor on GPU during forward, either precompute it at model init or start of forward pass.\n\n### Vision/Audio Model Pitfalls\n\n6. **Vision encoder attention must be bidirectional (non-causal)**:  `Sdpa.run_attention` with `flash_params: None` defaults to `causal = seq_len > 1` on the CUDA flash-attn path, which silently breaks vision/audio encoders. Always pass `FlashParams { causal: false, cumulative_seqlens_q: HashMap::new(), cumulative_seqlens_k: HashMap::new(), max_q: 0, max_k: 0 }` with `Some(&flash_params)` for any encoder that needs bidirectional attention. The empty `cumulative_seqlens` cause the flash backend to use the non-varlen kernel path, avoiding any tensor allocation in the forward pass.\n\n7. **`torch.bucketize(right=True)` requires `Ok(i) => i + 1`**: Rust's `binary_search_by` returns `Ok(i)` at the found position (bisect_left semantics). For `right=True` (bisect_right), you must use `Ok(i) => i + 1` to insert after equal elements. `Err(i) => i` is correct for both.\n\n8. **Mistral `consolidated.safetensors` stores Q/K weights with interleaved head dimensions**: When loading from Mistral-native `consolidated.safetensors` (as opposed to HF-converted `model.safetensors`), the Q and K projection weights use an interleaved layout within each head: `[x0, x_{d/2}, x1, x_{d/2+1}, ...]` instead of the sequential HF layout `[x0, x1, ..., x_{d/2-1}, x_{d/2}, ...]`. This means you must use `is_gptx=false` (GPT-J/adjacent-pair style) for `RotaryEmbedding`, NOT `is_gptx=true` (GPT-NeoX/half-split style). Using the wrong RoPE style produces completely wrong attention outputs (cosine similarity ~0.02 with reference). To diagnose: compare a Q or K weight tensor between `consolidated.safetensors` and `model.safetensors` — if they differ (cosine ~0.02), apply the un-interleave: `reshape(n_heads, head_dim/2, 2, dim).permute(0,2,1,3)` and verify cosine ~1.0.\n\n9. **Causal Conv1d padding formula**: For causal convolution (left-pad only, no right-pad), the correct left padding is `effective_kernel_size - stride`, NOT `(kernel_size - 1) * dilation` (which is the total padding for non-causal). For example, with kernel_size=3, stride=2, dilation=1: left_pad = 3 - 2 = 1, not 2. Verify against the HF model's `VoxtralRealtimeCausalConv1d` or equivalent source.\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","title":"AGENTS.md","content":"<!-- AGENTS.md: Guidance for AI agents to navigate, build, test, and contribute to this repository -->\n# AGENTS\n\nThis file provides instructions for AI agents to understand the layout of the `mistral.rs` repository, run builds/tests, and follow project conventions.\n\n## Repository Structure\n\n- `/mistralrs/`           : Main Rust crate (text & multimodal inference API)\n- `/mistralrs-core/`      : Core inference logic and tensor operations (text models)\n- `/mistralrs-vision/`    : Image processing utilities (resizing, preprocessing for multimodal models)\n- `/mistralrs-quant/`     : Quantization support (ISQ, GGUF, GPTQ, AWQ, FP8, HQQ, etc.)\n- `/mistralrs-paged-attn/`: PagedAttention implementation\n- `/mistralrs-pyo3/`      : Python bindings (PyO3)\n- `/mistralrs-cli/`       : Unified CLI binary (commands: run, serve, bench, from-config)\n- `/mistralrs-server-core/`: Shared server core logic\n- `/docs/`             : Astro/Starlight documentation site (deployed to GitHub Pages)\n- `/examples/`            : Usage examples (Rust, Python, server samples, notebooks)\n- `/chat_templates/`      : Chat formatting templates (JSON/Jinja)\n- `/scripts/`             : Utility scripts (e.g., AWQ conversion)\n  \n## Feature Organization\n\nMistral.rs supports multiple model types and advanced features via dedicated crates and CLI subcommands:\n\n- **Text Inference**\n  - Crate: `mistralrs-core` (low-level ops), `mistralrs` (API wrapper)\n  - CLI: `mistralrs run -m <model>` or `mistralrs serve -m <model>` (auto-detects model type)\n  - Docs: `docs/src/content/docs/guides/customize/sampling.md`, `docs/src/content/docs/guides/agents/`\n- **Multimodal Models**\n  - Crate: `mistralrs-vision`\n  - CLI: `mistralrs run -m <model>` (auto-detects multimodal models)\n  - Docs: `docs/src/content/docs/explanation/multimodal-pipeline.md`, `docs/src/content/docs/reference/supported-models.md`\n- **Diffusion Models**\n  - CLI: `mistralrs run -m <model>` (auto-detects diffusion models)\n  - Docs: `docs/src/content/docs/reference/supported-models.md`\n- **Speech Models**\n  - CLI: `mistralrs run -m <model>` (auto-detects speech models)\n  - Docs: `docs/src/content/docs/reference/supported-models.md`\n- **Quantization & ISQ**\n  - Crate: `mistralrs-quant`\n  - Docs: `docs/src/content/docs/reference/quantization-types.md`, `docs/src/content/docs/explanation/quantization-tradeoffs.md`\n  - Conversion Script: `scripts/convert_awq_marlin.py`\n- **Paged Attention**\n  - Crate: `mistralrs-paged-attn`\n  - Docs: `docs/src/content/docs/explanation/paged-attention.md`, `docs/src/content/docs/guides/perf/use-paged-attention.md`\n- **Adapters & LoRA/X-LoRA**\n  - Docs: `docs/src/content/docs/guides/customize/lora-adapters.md`\n- **Mixture of Experts (AnyMoE)**\n  - Docs: `docs/src/content/docs/guides/customize/anymoe.md`\n\n## Building\n\n1. Install Rust via rustup (Rust 2021 edition).\n2. Choose optional features (e.g., `cuda`, `flash-attn`, `cudnn`, `metal`, `mkl`, `accelerate`).\n3. Build the entire workspace:\n   ```bash\n   cargo build --workspace --release --features \"<features>\"\n   ```\n4. Or build/install only the CLI binary:\n   ```bash\n   cargo build --release --package mistralrs-cli --features \"<features>\"\n   cargo install --path mistralrs-cli --features \"<features>\"\n   ```\n\n## Models\n\nWhen integrating a new model, make sure it respects all of the varbuilder `.pp` calls. In Candle, a VarBuilder maintains an internal path vector that acts like a “current working directory” for model weights; every call to pp(\"sub\") (alias for push_prefix) clones the builder and appends sub, so successive calls accumulate a dotted prefix such as transformer.h.0 while leaving the original builder untouched . When you eventually call get(...), Candle joins that prefix with the tensor name (prefix + \".\" + name) and looks it up in the checkpoint backend, producing keys that exactly match the dot-separated names emitted by PyTorch’s state_dict/named_parameters, which means PyTorch-trained weights can be loaded without any renaming  . This lets you recreate the PyTorch module tree in Rust by “walking” it: e.g. vb.pp(\"word_embeddings\") grabs word_embeddings.*, while a chain like vb.pp(\"encoder\").pp(\"layers\").pp(i.to_string()) targets keys such as encoder.layers.0.*, exactly as shown in community tutorials porting Transformers models to Candle  . As one maintainer put it, the prefix system lets you “cd” around the parameter hierarchy, giving a lightweight namespace mechanism that keeps Candle fully compatible with PyTorch naming conventions while remaining ergonomic to use.\n\nYou should also look for a model.safetensors.index.json file for the model at hand to verify correct structure.\n\n## Testing\n\n- Core test suite (requires HF token for some tests):\n  ```bash\n  export HF_TOKEN=<your_token>  # or TESTS_HF_TOKEN for CI parity\n  cargo test -p mistralrs-core -p mistralrs-quant -p mistralrs-vision\n  ```\n- Run all tests across workspace (may skip some crates without tests):\n  ```bash\n  cargo test --workspace\n  ```\n\nYou should *always* run `cargo check`/`cargo c` before returning to make sure code compiles. If code does not compile, only make edits.\n\nAvoid returning TODOs.\n\n## Formatting & Linting\n\n- Format all Rust code:\n  ```bash\n  cargo fmt --all\n  make fmt       # also formats Python/CUDA/C++ files via ruff, clang-format\n  ```\n- Lint with Clippy:\n  ```bash\n  cargo clippy --workspace --tests --examples -- -D warnings\n  ```\n\n## Documentation\n\n- Generate Rust docs for all crates:\n  ```bash\n  cargo doc --workspace\n  ```\n- Preview Rust API docs at `target/doc/`.\n- Refer to `/docs/src/content/docs/` for in-depth guides. The site builds with `cd docs && npm run build` and deploys to GitHub Pages via `.github/workflows/docs.yml`.\n\n## Examples\n\n- Rust examples: `mistralrs/examples/`\n- Python examples: `examples/python/`\n- Server samples: `examples/server/`\n- Run Python scripts:\n  ```bash\n  python3 examples/python/<script>.py\n  ```\n- Run CLI:\n  ```bash\n  mistralrs run -m <model>        # Interactive mode\n  mistralrs serve -p 1234 -m <model>  # Server mode\n  mistralrs bench -m <model>      # Benchmarking\n  ```\n\n## CI Parity\n\nThe CI pipeline is defined in `.github/workflows/ci.yml` and includes:\n  - `cargo check` for all targets\n  - `cargo test` on core crates\n  - `cargo fmt -- --check`\n  - `cargo clippy -D warnings`\n  - `cargo doc`\n  - Typos check (`crate-ci/typos`)\n\n## Contribution Conventions\n\n- Follow Rust 2021 idioms, keep code minimal and focused.\n- Update `/docs/src/content/docs/` and examples when adding features or breaking changes.\n- Add tests and examples for new functionality.\n- Commit messages should be clear and follow conventional style where possible.\n  ```\n  feat(crate): describe new feature\n  fix(crate): describe bug fix\n  docs: update docs for ...\n  ```\n\n### Code Style (Extremely important & convention for this codebase)\n\n**Comments.** Default to none. Only add when the *why* isn't obvious from the code: hidden constraints, invariants, surprising edge cases, references to a spec/HF source. Never paraphrase what the next line does, never restate the function name, never narrate steps.\n\n- Multi-line comments are discouraged in code, and only really allowed in documentation or where they are the best way to communicate information.\n- Code comments should be one line each, up to ~120 cols. No multi-paragraph `///` blocks, no bulleted lists in doc comments, no `// === Section ===` or `// ── Section ──` banners.\n- Tone for inline code comments should be terse, casual, and never explaining what the code directly below does.\n- Only include code comments if they add new information, and never just for the sake of it.\n\n- Unless otherwise instructed, use ASCII only. No em-dashes (`—`), en-dashes (`–`), ellipses (`…`), smart quotes, or box-drawing characters. Do not use `--`. It's ok to use `...`, `\"`, `'` when appropriate.\n- Don't reference the current task / PR / fix / commit in comments — that belongs in the PR description and rots as the codebase evolves.\n- Trailing inline annotations like `// already sent above` are fine when terse.\n\n**Magic values.** Hoist durations, sizes, sentinels, and other constants to named `const`s at the top of the file. A sentinel value that crosses module boundaries (e.g. one place sets `Some(0)`, another checks for it) must be a `pub const`, not a literal both sides happen to share.\n\n**Function shape.** When a function passes 6+ args, prefer wrapping the invariants in a small context struct (e.g. `DispatchCtx<'a>`). Don't add error handling, fallbacks, or validation for scenarios that can't actually occur — trust internal code and framework guarantees. Don't add backwards-compatibility shims unless explicitly asked.\n\n---\n*This AGENTS.md file is intended solely to improve AI-driven assistance and does not affect runtime behavior.*\n","category":"root","tokens":2205},{"name":"CLAUDE.md","path":"CLAUDE.md","title":"CLAUDE.md","content":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nmistral.rs is a blazing-fast LLM inference engine written in Rust. It supports text, multimodal, image generation, and speech models with Rust and Python SDKs, plus OpenAI HTTP and MCP APIs.\n\n## Essential Commands\n\n### Building\n```bash\n# Basic release build\ncargo build --release\n\n# With CUDA support (Linux)\ncargo build --release --features \"cuda flash-attn cudnn\"\n\n# With Metal support (macOS)\ncargo build --release --features metal\n\n# Install CLI binary\ncargo install --path mistralrs-cli --features <features>\n```\n\n### Testing & Quality\n```bash\n# Run core tests\ncargo test -p mistralrs-core -p mistralrs-quant -p mistralrs-vision\n\n# Format code (uses rustfmt, ruff, clang-format)\nmake fmt\n\n# Check formatting\ncargo fmt --all -- --check\n\n# Run clippy\ncargo clippy --workspace --tests --examples -- -D warnings\n```\n\n### Running Models\n```bash\n# Run interactive mode (model type auto-detected)\nmistralrs run -m <model_id>\n\n# Run with GGUF quantized model\nmistralrs run --format gguf -m <repo> -f <file>\n\n# Run server\nmistralrs serve -p 1234 -m <model_id>\n\n# Run server (built-in web UI is on by default at /ui; pass --no-ui to disable)\nmistralrs serve -m <model_id>\n\n# Run benchmarks\nmistralrs bench -m <model_id>\n```\n\n## Models\n\nWhen integrating a new model, make sure it respects all of the varbuilder `.pp` calls. In Candle, a VarBuilder maintains an internal path vector that acts like a “current working directory” for model weights; every call to pp(\"sub\") (alias for push_prefix) clones the builder and appends sub, so successive calls accumulate a dotted prefix such as transformer.h.0 while leaving the original builder untouched . When you eventually call get(...), Candle joins that prefix with the tensor name (prefix + \".\" + name) and looks it up in the checkpoint backend, producing keys that exactly match the dot-separated names emitted by PyTorch’s state_dict/named_parameters, which means PyTorch-trained weights can be loaded without any renaming  . This lets you recreate the PyTorch module tree in Rust by “walking” it: e.g. vb.pp(\"word_embeddings\") grabs word_embeddings.*, while a chain like vb.pp(\"encoder\").pp(\"layers\").pp(i.to_string()) targets keys such as encoder.layers.0.*, exactly as shown in community tutorials porting Transformers models to Candle  . As one maintainer put it, the prefix system lets you “cd” around the parameter hierarchy, giving a lightweight namespace mechanism that keeps Candle fully compatible with PyTorch naming conventions while remaining ergonomic to use.\n\nYou should also look for a model.safetensors.index.json file for the model at hand to verify correct structure.\n\n## Architecture Overview\n\n### Workspace Structure\n- `mistralrs-core/` - Core inference engine, model implementations, pipelines\n- `mistralrs-cli/` - Unified CLI binary (commands: run, serve, bench, from-config)\n- `mistralrs-server-core/` - HTTP server routing, OpenAI API implementation\n- `mistralrs-pyo3/` - Python SDK (PyO3 bindings)\n- `mistralrs/` - Rust SDK (high-level crate)\n- `mistralrs-vision/` - Image processing utilities\n- `mistralrs-quant/` - Quantization implementations (ISQ, GGUF, GPTQ, etc.)\n- `mistralrs-paged-attn/` - PagedAttention implementation\n- `mistralrs-audio/` - Audio processing\n- `mistralrs-mcp/` - Model Context Protocol client\n\n### Key Design Patterns\n\n1. **Pipeline Architecture**: All models implement the `Pipeline` trait in `mistralrs-core/src/pipeline/mod.rs`. Different model types (Plain, GGUF, GGML, Multimodal) have their own pipeline implementations.\n\n2. **Model Loading**: Models are loaded through `Loader` traits that handle different formats and quantizations. See `mistralrs-core/src/loader.rs`.\n\n3. **Request Handling**: The server uses message passing with `MistralRs` struct managing a background thread pool. Requests flow through `mistralrs-core/src/engine/mod.rs`.\n\n4. **Device Management**: Automatic and manual device mapping for multi-GPU setups handled in `mistralrs-core/src/device_map.rs`.\n\n### Adding New Features\n\nWhen adding new model architectures:\n1. Implement the model in `mistralrs-core/src/models/`\n2. Add pipeline support in `mistralrs-core/src/pipeline/`\n3. Update model detection in `mistralrs-core/src/pipeline/normal.rs`\n4. Add architecture enum variant in `mistralrs-core/src/lib.rs`\n5. Update CLI args in `mistralrs-cli/src/main.rs`\n\nWhen adding new quantization methods:\n1. Implement in `mistralrs-quant/src/`\n2. Add to quantization loading logic in pipelines\n3. Update documentation in `docs/src/content/docs/reference/quantization-types.md`\n\n### Important Files to Know\n\n- `mistralrs-core/src/engine/mod.rs` - Main engine orchestration\n- `mistralrs-core/src/pipeline/mod.rs` - Pipeline trait and common logic\n- `mistralrs-server-core/src/routes.rs` - HTTP API endpoints\n- `mistralrs-pyo3/src/lib.rs` - Python SDK entry point\n- `mistralrs/examples/` - Usage examples for Rust SDK\n\n### Pull Requests\n\nNever include a \"Test plan\" section in PR descriptions.\n\n### Code Style (Extremely important & convention for this codebase)\n\n**Comments.** Default to none. Only add when the *why* isn't obvious from the code: hidden constraints, invariants, surprising edge cases, references to a spec/HF source. Never paraphrase what the next line does, never restate the function name, never narrate steps.\n\n- Multi-line comments are discouraged in code, and only really allowed in documentation or where they are the best way to communicate information.\n- Code comments should be one line each, up to ~120 cols. No multi-paragraph `///` blocks, no bulleted lists in doc comments, no `// === Section ===` or `// ── Section ──` banners.\n- Tone for inline code comments should be terse, casual, and never explaining what the code directly below does.\n- Only include code comments if they add new information, and never just for the sake of it.\n\n- Unless otherwise instructed, use ASCII only. No em-dashes (`—`), en-dashes (`–`), ellipses (`…`), smart quotes, or box-drawing characters. Do not use `--`. It's ok to use `...`, `\"`, `'` when appropriate.\n- Don't reference the current task / PR / fix / commit in comments — that belongs in the PR description and rots as the codebase evolves.\n- Trailing inline annotations like `// already sent above` are fine when terse.\n\n**Magic values.** Hoist durations, sizes, sentinels, and other constants to named `const`s at the top of the file. A sentinel value that crosses module boundaries (e.g. one place sets `Some(0)`, another checks for it) must be a `pub const`, not a literal both sides happen to share.\n\n**Function shape.** When a function passes 6+ args, prefer wrapping the invariants in a small context struct (e.g. `DispatchCtx<'a>`). Don't add error handling, fallbacks, or validation for scenarios that can't actually occur — trust internal code and framework guarantees. Don't add backwards-compatibility shims unless explicitly asked.\n\n### Testing Approach\n\nYou should *always* run `cargo check`/`cargo c` before returning to make sure code compiles. If code does not compile, only make edits.\n\nAvoid returning TODOs.\n\n- Unit tests are colocated with source files\n- Integration tests in `tests/` directories\n- Use `cargo test -p <crate>` to test specific components\n- Python tests require building and installing the package first\n\n### Common Pitfalls\n\n1. **Feature Flags**: Many features are gated behind Cargo features. Always check what features are needed for your use case.\n2. **Device Indices**: CUDA device selection uses 0-based indexing\n3. **Chat Templates**: Models may need specific chat templates - check `chat_templates/` directory\n4. **Quantization**: Different quantization methods have different hardware requirements\n5. **Never use `Tensor::{from_vec,arange}` in hot loops**: `Tensor::{from_vec,arange}` with a GPU device causes a CPU-to-GPU sync. If you need a small tensor on GPU during forward, either precompute it at model init or start of forward pass.\n\n### Vision/Audio Model Pitfalls\n\n6. **Vision encoder attention must be bidirectional (non-causal)**:  `Sdpa.run_attention` with `flash_params: None` defaults to `causal = seq_len > 1` on the CUDA flash-attn path, which silently breaks vision/audio encoders. Always pass `FlashParams { causal: false, cumulative_seqlens_q: HashMap::new(), cumulative_seqlens_k: HashMap::new(), max_q: 0, max_k: 0 }` with `Some(&flash_params)` for any encoder that needs bidirectional attention. The empty `cumulative_seqlens` cause the flash backend to use the non-varlen kernel path, avoiding any tensor allocation in the forward pass.\n\n7. **`torch.bucketize(right=True)` requires `Ok(i) => i + 1`**: Rust's `binary_search_by` returns `Ok(i)` at the found position (bisect_left semantics). For `right=True` (bisect_right), you must use `Ok(i) => i + 1` to insert after equal elements. `Err(i) => i` is correct for both.\n\n8. **Mistral `consolidated.safetensors` stores Q/K weights with interleaved head dimensions**: When loading from Mistral-native `consolidated.safetensors` (as opposed to HF-converted `model.safetensors`), the Q and K projection weights use an interleaved layout within each head: `[x0, x_{d/2}, x1, x_{d/2+1}, ...]` instead of the sequential HF layout `[x0, x1, ..., x_{d/2-1}, x_{d/2}, ...]`. This means you must use `is_gptx=false` (GPT-J/adjacent-pair style) for `RotaryEmbedding`, NOT `is_gptx=true` (GPT-NeoX/half-split style). Using the wrong RoPE style produces completely wrong attention outputs (cosine similarity ~0.02 with reference). To diagnose: compare a Q or K weight tensor between `consolidated.safetensors` and `model.safetensors` — if they differ (cosine ~0.02), apply the un-interleave: `reshape(n_heads, head_dim/2, 2, dim).permute(0,2,1,3)` and verify cosine ~1.0.\n\n9. **Causal Conv1d padding formula**: For causal convolution (left-pad only, no right-pad), the correct left padding is `effective_kernel_size - stride`, NOT `(kernel_size - 1) * dilation` (which is the total padding for non-causal). For example, with kernel_size=3, stride=2, dilation=1: left_pad = 3 - 2 = 1, not 2. Verify against the HF model's `VoxtralRealtimeCausalConv1d` or equivalent source.\n","category":"root","tokens":2565}]}