{"owner":"mistralai","repo":"mistral-common","hasSkills":true,"totalSkillsCount":1,"totalTokensCount":1985,"categories":["root-instruction"],"hasMcp":false,"mcpConfig":null,"found":["AGENTS.md"],"skills":{"AGENTS.md":"# AGENTS.md\n\n## Project overview\n\nMistral-Common is a preprocessing library for Mistral's Large Language Models (LLMs).\nIt encodes requests for Instruct, Transcription or Fill-In-The-Middle (FIM) tasks to tokens and optionally processed images or audios.\n- Language: Python 3.10 to 3.15\n- Package Manager: uv\n- Testing: pytest\n- Formatting and Linting: Ruff\n- Type checker: mypy\n- CI: GitHub Actions\n\n## Project Structure\n\n```\nmistral-common/\n├── src/\n│   └── mistral_common/\n│       ├── guidance/\n│       ├── integrations/\n│       ├── protocol/\n│       ├── tokens/\n│       └── ...\n├── scripts/\n├── tests/\n├── docs/\n├── .github/\n├── .pre-commit-config.yaml\n├── pyproject.toml\n└── README.md\n```\n\n### Root-level files in src/mistral_common/\n- `audio.py`: Audio processing utilities including Audio class and mel-scale conversions\n- `base.py`: Base Pydantic model configuration\n- `deprecation.py`: Deprecation utilities (`deprecated_import`, `warn_once`) for emitting one-shot warnings on moved or removed symbols\n- `exceptions.py`: Custom exception classes for the library\n- `image.py`: Image processing utilities including download and serialization\n- `imports.py`: Import utilities and dependency checks\n- `multimodal.py`: Multimodal processing utilities deprecated in favor to `image.py`\n\n### Protocol\n- `src/mistral_common/protocol/`: Protocol handling\n  - `instruct/`: Instruct protocol\n    - `chunk.py`: Chunks content used by messages\n    - `converters.py`: Converter helpers between `ChatCompletionRequest` and `Openai` requests\n    - `messages.py`: Instruct messages definition\n    - `normalize.py`: Normalizers for `ChatCompletionRequest`\n    - `request.py`: Definition of `ChatCompletionRequest`. This is the entry point of user queries for Instruct requests\n    - `tool_calls.py`: Tool calling logic\n    - `validator.py`: Validators for `ChatCompletionRequest`\n  - `fim/`: Fill-in-the-middle protocol\n    - `request.py`: Definition of `FIMRequest`. This is the entry point of user queries for FIM requests\n  - `transcription/`: Transcription protocol\n    - `request.py`: Definition of `TranscriptionRequest`. This is the entry point of user queries for Transcription requests\n  - `speech/`: Speech protocol\n    - `request.py`: Definition of `SpeechRequest`. This is the entry point of user queries for Speech requests\n  - `base.py`: Definition of `BaseCompletionRequest` subclassed by FIM and Instruct requests\n  - `utils.py`: Utility functions\n\n### Tokenization\n- `src/mistral_common/tokens/`: Tokenization\n  - `tokenizers/`: Tokenizer implementations\n    - `audio.py`: Audio processing\n    - `base.py`: Base Tokenizer implementation\n    - `image.py`: Image processing\n    - `instruct.py`: Instruct Tokenizer that encodes requests via Tekken tokenizer\n    - `mistral.py`: Mistral Tokenizer that normalizes and validates requests to pass them to an instruct tokenizer\n    - `model_settings_builder.py`: Builders (`FieldBuilder`, `EnumBuilder`, `ModelSettingsBuilder`) for validating and constructing model settings\n    - `multimodal.py`: deprecated in favor of `image.py`\n    - `sentencepiece.py`: Sentence Piece tokenizer (deprecated)\n    - `tekken.py`: Tekken tokenizer used by all recent models\n    - `utils.py`: Utility functions for the tokenizers\n  - `instruct/`: deprecated in favor of `src/mistral_common/protocol/instruct/request.py`\n\n### Guidance (Grammar)\n- `src/mistral_common/guidance/`: Creates Lark grammars for tool calls, JSON schema and reasoning using llguidance\n  - `grammar_factory.py`: `GrammarFactory` that builds and renders Lark grammars from Jinja templates\n  - `tokenizer.py`: Adapts Tekken tokenizer for llguidance\n  - `data/`: Jinja-templated Lark grammar files for base, thinking (special tokens) and thinking (plain text) modes\n\n### Integrations\n- `src/mistral_common/integrations/`: Third-party framework integrations\n  - `chat_templates/`: Chat template generation for HuggingFace Transformers\n    - `chat_templates.py`: Public API for generating chat templates (`generate_chat_template`)\n    - `template_generator.py`: Core template generation engine with `TemplateConfig` and `build_chat_template`\n\n### Scripts\n- `scripts/generate_chat_template.py`: CLI for generating and saving chat templates\n\n## Experimental\n- `src/mistral_common/experimental/`: Experimental features\n  - `utils.py`: Utility functions\n  - `tools.py`: Tool calls parser\n  - `think.py`: Thinking parser\n  - `app/`: FastAPI application\n    - `routers.py`: API routers\n    - `main.py`: Application entry point\n    - `models.py`: Pydantic models\n\n## Data\n- `src/mistral_common/data/`: Data files for tokenizers\n\n### Other files and directories\n- `tests/`: Test suite\n- `docs/`: Documentation\n- `.github/workflows/`: CI/CD workflows\n- `.pre-commit-config.yaml`: Pre-commit hooks\n- `pyproject.toml`: Project configuration\n\n## Code Style Guidelines\n\n### Style\n- Respect ruff and mypy rules\n- Naming: snake_case for functions/variables, PascalCase for classes\n- Use Python functionalities supported by Python 3.10\n- Call function arguments explicitly by keyword, not implicitly by position (e.g. `fn(x=1, y=2)`, not `fn(1, 2)`)\n\n### Comments\n- Do NOT write comments that paraphrase or restate what the code already says.\n- Comments should explain the \"why\" (intent, rationale, non-obvious constraints), not the \"what\".\n\n### Imports\n- Use absolute imports for modules within the project\n- Do NOT use wildcard imports\n- Do NOT add import inside `__init__`\n- Use `TYPE_CHECKING` blocks for type-only imports\n- Do NOT use `from __future__ import annotations`\n\n### Type Hints (Required)\n- Use Python's type hints extensively\n- Use modern (Python 3.10+) typing module types\n\n### Error Handling\n- Use custom exceptions from `mistral_common.exceptions`\n- Provide meaningful error messages\n\n### Docstrings (Required)\n- Use Google-style docstrings\n- One-liner for simple functions\n- Multi-line with Args/Returns for complex ones\n- Use `r\"\"\"` for raw docstrings\n- Document all parameters and return values.\n  - Do NOT put types in the docstring for parameters and return values.\n  - For the returns sections, only describe the returned value and do not write its name\n- Include examples where appropriate that can be tested via `doctest`\n\n### Coverage\n- Aim for high, meaningful coverage. Prioritise tests that verify behaviour over hitting a line-count target.\n- New and changed code should be covered by tests.\n- Avoid coverage-only comments. Prefer restructuring so branches are genuinely reachable and tested (e.g. validate inputs and test the error path) over excluding lines.\n\n### Test organization\n\n- New unit tests belong under `tests/unit/` and mirror the source package structure.\n- New integration tests belong under `tests/integration/` and exercise public workflows.\n- Existing tests elsewhere are legacy during the ongoing migration and are removed as their replacements land.\n\n## Development Workflow\n\n1. Set up using uv\n\n```bash\nuv sync --frozen --all-extras --group dev --python 3.12\nsource .venv/bin/activate\nuv run pre-commit install\n```\n\n2.  Make Changes\n- Follow code style guidelines\n- Write tests for new functionality\n- Update documentation\n- When adding dependencies, modify root `pyproject.toml`, then run `uv lock` followed by `uv sync --frozen`\n- Backward Compatibility: Don't break existing functionality\n\n3. Run linter, formatter (Ruff), type checker (mypy) and tests (pytest), including doctests\n\n### Build documentation locally\n\nThe documentation is built with [Zensical](https://zensical.org) (a drop-in successor to MkDocs that reads `mkdocs.yml` natively):\n\n```bash\nuv sync --frozen --all-extras --group docs\nuv run python docs/gen_api_pages.py  # generate the code reference pages\nuv run zensical serve                # live preview at http://localhost:8000\n```\n\nUse `uv run zensical build` for a one-shot static build into `site/`.\n\n### Commit\n- After adding your changes before committing ensure pre-commit is installed or run it manually.\n- Use imperative grammar, start with a verb and be concise.\n\n## Additional Resources\n\n- [Pydantic Documentation](https://docs.pydantic.dev/latest/)\n- [Ruff Documentation](https://docs.astral.sh/ruff/)\n- [Mypy Documentation](https://mypy.readthedocs.io/)\n- [Pytest Documentation](https://docs.pytest.org/)\n"},"files":{"AGENTS.md":"# AGENTS.md\n\n## Project overview\n\nMistral-Common is a preprocessing library for Mistral's Large Language Models (LLMs).\nIt encodes requests for Instruct, Transcription or Fill-In-The-Middle (FIM) tasks to tokens and optionally processed images or audios.\n- Language: Python 3.10 to 3.15\n- Package Manager: uv\n- Testing: pytest\n- Formatting and Linting: Ruff\n- Type checker: mypy\n- CI: GitHub Actions\n\n## Project Structure\n\n```\nmistral-common/\n├── src/\n│   └── mistral_common/\n│       ├── guidance/\n│       ├── integrations/\n│       ├── protocol/\n│       ├── tokens/\n│       └── ...\n├── scripts/\n├── tests/\n├── docs/\n├── .github/\n├── .pre-commit-config.yaml\n├── pyproject.toml\n└── README.md\n```\n\n### Root-level files in src/mistral_common/\n- `audio.py`: Audio processing utilities including Audio class and mel-scale conversions\n- `base.py`: Base Pydantic model configuration\n- `deprecation.py`: Deprecation utilities (`deprecated_import`, `warn_once`) for emitting one-shot warnings on moved or removed symbols\n- `exceptions.py`: Custom exception classes for the library\n- `image.py`: Image processing utilities including download and serialization\n- `imports.py`: Import utilities and dependency checks\n- `multimodal.py`: Multimodal processing utilities deprecated in favor to `image.py`\n\n### Protocol\n- `src/mistral_common/protocol/`: Protocol handling\n  - `instruct/`: Instruct protocol\n    - `chunk.py`: Chunks content used by messages\n    - `converters.py`: Converter helpers between `ChatCompletionRequest` and `Openai` requests\n    - `messages.py`: Instruct messages definition\n    - `normalize.py`: Normalizers for `ChatCompletionRequest`\n    - `request.py`: Definition of `ChatCompletionRequest`. This is the entry point of user queries for Instruct requests\n    - `tool_calls.py`: Tool calling logic\n    - `validator.py`: Validators for `ChatCompletionRequest`\n  - `fim/`: Fill-in-the-middle protocol\n    - `request.py`: Definition of `FIMRequest`. This is the entry point of user queries for FIM requests\n  - `transcription/`: Transcription protocol\n    - `request.py`: Definition of `TranscriptionRequest`. This is the entry point of user queries for Transcription requests\n  - `speech/`: Speech protocol\n    - `request.py`: Definition of `SpeechRequest`. This is the entry point of user queries for Speech requests\n  - `base.py`: Definition of `BaseCompletionRequest` subclassed by FIM and Instruct requests\n  - `utils.py`: Utility functions\n\n### Tokenization\n- `src/mistral_common/tokens/`: Tokenization\n  - `tokenizers/`: Tokenizer implementations\n    - `audio.py`: Audio processing\n    - `base.py`: Base Tokenizer implementation\n    - `image.py`: Image processing\n    - `instruct.py`: Instruct Tokenizer that encodes requests via Tekken tokenizer\n    - `mistral.py`: Mistral Tokenizer that normalizes and validates requests to pass them to an instruct tokenizer\n    - `model_settings_builder.py`: Builders (`FieldBuilder`, `EnumBuilder`, `ModelSettingsBuilder`) for validating and constructing model settings\n    - `multimodal.py`: deprecated in favor of `image.py`\n    - `sentencepiece.py`: Sentence Piece tokenizer (deprecated)\n    - `tekken.py`: Tekken tokenizer used by all recent models\n    - `utils.py`: Utility functions for the tokenizers\n  - `instruct/`: deprecated in favor of `src/mistral_common/protocol/instruct/request.py`\n\n### Guidance (Grammar)\n- `src/mistral_common/guidance/`: Creates Lark grammars for tool calls, JSON schema and reasoning using llguidance\n  - `grammar_factory.py`: `GrammarFactory` that builds and renders Lark grammars from Jinja templates\n  - `tokenizer.py`: Adapts Tekken tokenizer for llguidance\n  - `data/`: Jinja-templated Lark grammar files for base, thinking (special tokens) and thinking (plain text) modes\n\n### Integrations\n- `src/mistral_common/integrations/`: Third-party framework integrations\n  - `chat_templates/`: Chat template generation for HuggingFace Transformers\n    - `chat_templates.py`: Public API for generating chat templates (`generate_chat_template`)\n    - `template_generator.py`: Core template generation engine with `TemplateConfig` and `build_chat_template`\n\n### Scripts\n- `scripts/generate_chat_template.py`: CLI for generating and saving chat templates\n\n## Experimental\n- `src/mistral_common/experimental/`: Experimental features\n  - `utils.py`: Utility functions\n  - `tools.py`: Tool calls parser\n  - `think.py`: Thinking parser\n  - `app/`: FastAPI application\n    - `routers.py`: API routers\n    - `main.py`: Application entry point\n    - `models.py`: Pydantic models\n\n## Data\n- `src/mistral_common/data/`: Data files for tokenizers\n\n### Other files and directories\n- `tests/`: Test suite\n- `docs/`: Documentation\n- `.github/workflows/`: CI/CD workflows\n- `.pre-commit-config.yaml`: Pre-commit hooks\n- `pyproject.toml`: Project configuration\n\n## Code Style Guidelines\n\n### Style\n- Respect ruff and mypy rules\n- Naming: snake_case for functions/variables, PascalCase for classes\n- Use Python functionalities supported by Python 3.10\n- Call function arguments explicitly by keyword, not implicitly by position (e.g. `fn(x=1, y=2)`, not `fn(1, 2)`)\n\n### Comments\n- Do NOT write comments that paraphrase or restate what the code already says.\n- Comments should explain the \"why\" (intent, rationale, non-obvious constraints), not the \"what\".\n\n### Imports\n- Use absolute imports for modules within the project\n- Do NOT use wildcard imports\n- Do NOT add import inside `__init__`\n- Use `TYPE_CHECKING` blocks for type-only imports\n- Do NOT use `from __future__ import annotations`\n\n### Type Hints (Required)\n- Use Python's type hints extensively\n- Use modern (Python 3.10+) typing module types\n\n### Error Handling\n- Use custom exceptions from `mistral_common.exceptions`\n- Provide meaningful error messages\n\n### Docstrings (Required)\n- Use Google-style docstrings\n- One-liner for simple functions\n- Multi-line with Args/Returns for complex ones\n- Use `r\"\"\"` for raw docstrings\n- Document all parameters and return values.\n  - Do NOT put types in the docstring for parameters and return values.\n  - For the returns sections, only describe the returned value and do not write its name\n- Include examples where appropriate that can be tested via `doctest`\n\n### Coverage\n- Aim for high, meaningful coverage. Prioritise tests that verify behaviour over hitting a line-count target.\n- New and changed code should be covered by tests.\n- Avoid coverage-only comments. Prefer restructuring so branches are genuinely reachable and tested (e.g. validate inputs and test the error path) over excluding lines.\n\n### Test organization\n\n- New unit tests belong under `tests/unit/` and mirror the source package structure.\n- New integration tests belong under `tests/integration/` and exercise public workflows.\n- Existing tests elsewhere are legacy during the ongoing migration and are removed as their replacements land.\n\n## Development Workflow\n\n1. Set up using uv\n\n```bash\nuv sync --frozen --all-extras --group dev --python 3.12\nsource .venv/bin/activate\nuv run pre-commit install\n```\n\n2.  Make Changes\n- Follow code style guidelines\n- Write tests for new functionality\n- Update documentation\n- When adding dependencies, modify root `pyproject.toml`, then run `uv lock` followed by `uv sync --frozen`\n- Backward Compatibility: Don't break existing functionality\n\n3. Run linter, formatter (Ruff), type checker (mypy) and tests (pytest), including doctests\n\n### Build documentation locally\n\nThe documentation is built with [Zensical](https://zensical.org) (a drop-in successor to MkDocs that reads `mkdocs.yml` natively):\n\n```bash\nuv sync --frozen --all-extras --group docs\nuv run python docs/gen_api_pages.py  # generate the code reference pages\nuv run zensical serve                # live preview at http://localhost:8000\n```\n\nUse `uv run zensical build` for a one-shot static build into `site/`.\n\n### Commit\n- After adding your changes before committing ensure pre-commit is installed or run it manually.\n- Use imperative grammar, start with a verb and be concise.\n\n## Additional Resources\n\n- [Pydantic Documentation](https://docs.pydantic.dev/latest/)\n- [Ruff Documentation](https://docs.astral.sh/ruff/)\n- [Mypy Documentation](https://mypy.readthedocs.io/)\n- [Pytest Documentation](https://docs.pytest.org/)\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/mistralai/mistral-common/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# AGENTS.md\n\n## Project overview\n\nMistral-Common is a preprocessing library for Mistral's Large Language Models (LLMs).\nIt encodes requests for Instruct, Transcription or Fill-In-The-Middle (FIM) tasks to tokens and optionally processed images or audios.\n- Language: Python 3.10 to 3.15\n- Package Manager: uv\n- Testing: pytest\n- Formatting and Linting: Ruff\n- Type checker: mypy\n- CI: GitHub Actions\n\n## Project Structure\n\n```\nmistral-common/\n├── src/\n│   └── mistral_common/\n│       ├── guidance/\n│       ├── integrations/\n│       ├── protocol/\n│       ├── tokens/\n│       └── ...\n├── scripts/\n├── tests/\n├── docs/\n├── .github/\n├── .pre-commit-config.yaml\n├── pyproject.toml\n└── README.md\n```\n\n### Root-level files in src/mistral_common/\n- `audio.py`: Audio processing utilities including Audio class and mel-scale conversions\n- `base.py`: Base Pydantic model configuration\n- `deprecation.py`: Deprecation utilities (`deprecated_import`, `warn_once`) for emitting one-shot warnings on moved or removed symbols\n- `exceptions.py`: Custom exception classes for the library\n- `image.py`: Image processing utilities including download and serialization\n- `imports.py`: Import utilities and dependency checks\n- `multimodal.py`: Multimodal processing utilities deprecated in favor to `image.py`\n\n### Protocol\n- `src/mistral_common/protocol/`: Protocol handling\n  - `instruct/`: Instruct protocol\n    - `chunk.py`: Chunks content used by messages\n    - `converters.py`: Converter helpers between `ChatCompletionRequest` and `Openai` requests\n    - `messages.py`: Instruct messages definition\n    - `normalize.py`: Normalizers for `ChatCompletionRequest`\n    - `request.py`: Definition of `ChatCompletionRequest`. This is the entry point of user queries for Instruct requests\n    - `tool_calls.py`: Tool calling logic\n    - `validator.py`: Validators for `ChatCompletionRequest`\n  - `fim/`: Fill-in-the-middle protocol\n    - `request.py`: Definition of `FIMRequest`. This is the entry point of user queries for FIM requests\n  - `transcription/`: Transcription protocol\n    - `request.py`: Definition of `TranscriptionRequest`. This is the entry point of user queries for Transcription requests\n  - `speech/`: Speech protocol\n    - `request.py`: Definition of `SpeechRequest`. This is the entry point of user queries for Speech requests\n  - `base.py`: Definition of `BaseCompletionRequest` subclassed by FIM and Instruct requests\n  - `utils.py`: Utility functions\n\n### Tokenization\n- `src/mistral_common/tokens/`: Tokenization\n  - `tokenizers/`: Tokenizer implementations\n    - `audio.py`: Audio processing\n    - `base.py`: Base Tokenizer implementation\n    - `image.py`: Image processing\n    - `instruct.py`: Instruct Tokenizer that encodes requests via Tekken tokenizer\n    - `mistral.py`: Mistral Tokenizer that normalizes and validates requests to pass them to an instruct tokenizer\n    - `model_settings_builder.py`: Builders (`FieldBuilder`, `EnumBuilder`, `ModelSettingsBuilder`) for validating and constructing model settings\n    - `multimodal.py`: deprecated in favor of `image.py`\n    - `sentencepiece.py`: Sentence Piece tokenizer (deprecated)\n    - `tekken.py`: Tekken tokenizer used by all recent models\n    - `utils.py`: Utility functions for the tokenizers\n  - `instruct/`: deprecated in favor of `src/mistral_common/protocol/instruct/request.py`\n\n### Guidance (Grammar)\n- `src/mistral_common/guidance/`: Creates Lark grammars for tool calls, JSON schema and reasoning using llguidance\n  - `grammar_factory.py`: `GrammarFactory` that builds and renders Lark grammars from Jinja templates\n  - `tokenizer.py`: Adapts Tekken tokenizer for llguidance\n  - `data/`: Jinja-templated Lark grammar files for base, thinking (special tokens) and thinking (plain text) modes\n\n### Integrations\n- `src/mistral_common/integrations/`: Third-party framework integrations\n  - `chat_templates/`: Chat template generation for HuggingFace Transformers\n    - `chat_templates.py`: Public API for generating chat templates (`generate_chat_template`)\n    - `template_generator.py`: Core template generation engine with `TemplateConfig` and `build_chat_template`\n\n### Scripts\n- `scripts/generate_chat_template.py`: CLI for generating and saving chat templates\n\n## Experimental\n- `src/mistral_common/experimental/`: Experimental features\n  - `utils.py`: Utility functions\n  - `tools.py`: Tool calls parser\n  - `think.py`: Thinking parser\n  - `app/`: FastAPI application\n    - `routers.py`: API routers\n    - `main.py`: Application entry point\n    - `models.py`: Pydantic models\n\n## Data\n- `src/mistral_common/data/`: Data files for tokenizers\n\n### Other files and directories\n- `tests/`: Test suite\n- `docs/`: Documentation\n- `.github/workflows/`: CI/CD workflows\n- `.pre-commit-config.yaml`: Pre-commit hooks\n- `pyproject.toml`: Project configuration\n\n## Code Style Guidelines\n\n### Style\n- Respect ruff and mypy rules\n- Naming: snake_case for functions/variables, PascalCase for classes\n- Use Python functionalities supported by Python 3.10\n- Call function arguments explicitly by keyword, not implicitly by position (e.g. `fn(x=1, y=2)`, not `fn(1, 2)`)\n\n### Comments\n- Do NOT write comments that paraphrase or restate what the code already says.\n- Comments should explain the \"why\" (intent, rationale, non-obvious constraints), not the \"what\".\n\n### Imports\n- Use absolute imports for modules within the project\n- Do NOT use wildcard imports\n- Do NOT add import inside `__init__`\n- Use `TYPE_CHECKING` blocks for type-only imports\n- Do NOT use `from __future__ import annotations`\n\n### Type Hints (Required)\n- Use Python's type hints extensively\n- Use modern (Python 3.10+) typing module types\n\n### Error Handling\n- Use custom exceptions from `mistral_common.exceptions`\n- Provide meaningful error messages\n\n### Docstrings (Required)\n- Use Google-style docstrings\n- One-liner for simple functions\n- Multi-line with Args/Returns for complex ones\n- Use `r\"\"\"` for raw docstrings\n- Document all parameters and return values.\n  - Do NOT put types in the docstring for parameters and return values.\n  - For the returns sections, only describe the returned value and do not write its name\n- Include examples where appropriate that can be tested via `doctest`\n\n### Coverage\n- Aim for high, meaningful coverage. Prioritise tests that verify behaviour over hitting a line-count target.\n- New and changed code should be covered by tests.\n- Avoid coverage-only comments. Prefer restructuring so branches are genuinely reachable and tested (e.g. validate inputs and test the error path) over excluding lines.\n\n### Test organization\n\n- New unit tests belong under `tests/unit/` and mirror the source package structure.\n- New integration tests belong under `tests/integration/` and exercise public workflows.\n- Existing tests elsewhere are legacy during the ongoing migration and are removed as their replacements land.\n\n## Development Workflow\n\n1. Set up using uv\n\n```bash\nuv sync --frozen --all-extras --group dev --python 3.12\nsource .venv/bin/activate\nuv run pre-commit install\n```\n\n2.  Make Changes\n- Follow code style guidelines\n- Write tests for new functionality\n- Update documentation\n- When adding dependencies, modify root `pyproject.toml`, then run `uv lock` followed by `uv sync --frozen`\n- Backward Compatibility: Don't break existing functionality\n\n3. Run linter, formatter (Ruff), type checker (mypy) and tests (pytest), including doctests\n\n### Build documentation locally\n\nThe documentation is built with [Zensical](https://zensical.org) (a drop-in successor to MkDocs that reads `mkdocs.yml` natively):\n\n```bash\nuv sync --frozen --all-extras --group docs\nuv run python docs/gen_api_pages.py  # generate the code reference pages\nuv run zensical serve                # live preview at http://localhost:8000\n```\n\nUse `uv run zensical build` for a one-shot static build into `site/`.\n\n### Commit\n- After adding your changes before committing ensure pre-commit is installed or run it manually.\n- Use imperative grammar, start with a verb and be concise.\n\n## Additional Resources\n\n- [Pydantic Documentation](https://docs.pydantic.dev/latest/)\n- [Ruff Documentation](https://docs.astral.sh/ruff/)\n- [Mypy Documentation](https://mypy.readthedocs.io/)\n- [Pytest Documentation](https://docs.pytest.org/)\n","isInternal":false,"tokens":1985,"sizeBytes":8375}],"systemPromptSnippet":"<yakaai_skills repo=\"mistralai/mistral-common\">\n<!-- File: AGENTS.md (Tokens: ~1985 | Category: root-instruction) -->\n# AGENTS.md\n\n## Project overview\n\nMistral-Common is a preprocessing library for Mistral's Large Language Models (LLMs).\nIt encodes requests for Instruct, Transcription or Fill-In-The-Middle (FIM) tasks to tokens and optionally processed images or audios.\n- Language: Python 3.10 to 3.15\n- Package Manager: uv\n- Testing: pytest\n- Formatting and Linting: Ruff\n- Type checker: mypy\n- CI: GitHub Actions\n\n## Project Structure\n\n```\nmistral-common/\n├── src/\n│   └── mistral_common/\n│       ├── guidance/\n│       ├── integrations/\n│       ├── protocol/\n│       ├── tokens/\n│       └── ...\n├── scripts/\n├── tests/\n├── docs/\n├── .github/\n├── .pre-commit-config.yaml\n├── pyproject.toml\n└── README.md\n```\n\n### Root-level files in src/mistral_common/\n- `audio.py`: Audio processing utilities including Audio class and mel-scale conversions\n- `base.py`: Base Pydantic model configuration\n- `deprecation.py`: Deprecation utilities (`deprecated_import`, `warn_once`) for emitting one-shot warnings on moved or removed symbols\n- `exceptions.py`: Custom exception classes for the library\n- `image.py`: Image processing utilities including download and serialization\n- `imports.py`: Import utilities and dependency checks\n- `multimodal.py`: Multimodal processing utilities deprecated in favor to `image.py`\n\n### Protocol\n- `src/mistral_common/protocol/`: Protocol handling\n  - `instruct/`: Instruct protocol\n    - `chunk.py`: Chunks content used by messages\n    - `converters.py`: Converter helpers between `ChatCompletionRequest` and `Openai` requests\n    - `messages.py`: Instruct messages definition\n    - `normalize.py`: Normalizers for `ChatCompletionRequest`\n    - `request.py`: Definition of `ChatCompletionRequest`. This is the entry point of user queries for Instruct requests\n    - `tool_calls.py`: Tool calling logic\n    - `validator.py`: Validators for `ChatCompletionRequest`\n  - `fim/`: Fill-in-the-middle protocol\n    - `request.py`: Definition of `FIMRequest`. This is the entry point of user queries for FIM requests\n  - `transcription/`: Transcription protocol\n    - `request.py`: Definition of `TranscriptionRequest`. This is the entry point of user queries for Transcription requests\n  - `speech/`: Speech protocol\n    - `request.py`: Definition of `SpeechRequest`. This is the entry point of user queries for Speech requests\n  - `base.py`: Definition of `BaseCompletionRequest` subclassed by FIM and Instruct requests\n  - `utils.py`: Utility functions\n\n### Tokenization\n- `src/mistral_common/tokens/`: Tokenization\n  - `tokenizers/`: Tokenizer implementations\n    - `audio.py`: Audio processing\n    - `base.py`: Base Tokenizer implementation\n    - `image.py`: Image processing\n    - `instruct.py`: Instruct Tokenizer that encodes requests via Tekken tokenizer\n    - `mistral.py`: Mistral Tokenizer that normalizes and validates requests to pass them to an instruct tokenizer\n    - `model_settings_builder.py`: Builders (`FieldBuilder`, `EnumBuilder`, `ModelSettingsBuilder`) for validating and constructing model settings\n    - `multimodal.py`: deprecated in favor of `image.py`\n    - `sentencepiece.py`: Sentence Piece tokenizer (deprecated)\n    - `tekken.py`: Tekken tokenizer used by all recent models\n    - `utils.py`: Utility functions for the tokenizers\n  - `instruct/`: deprecated in favor of `src/mistral_common/protocol/instruct/request.py`\n\n### Guidance (Grammar)\n- `src/mistral_common/guidance/`: Creates Lark grammars for tool calls, JSON schema and reasoning using llguidance\n  - `grammar_factory.py`: `GrammarFactory` that builds and renders Lark grammars from Jinja templates\n  - `tokenizer.py`: Adapts Tekken tokenizer for llguidance\n  - `data/`: Jinja-templated Lark grammar files for base, thinking (special tokens) and thinking (plain text) modes\n\n### Integrations\n- `src/mistral_common/integrations/`: Third-party framework integrations\n  - `chat_templates/`: Chat template generation for HuggingFace Transformers\n    - `chat_templates.py`: Public API for generating chat templates (`generate_chat_template`)\n    - `template_generator.py`: Core template generation engine with `TemplateConfig` and `build_chat_template`\n\n### Scripts\n- `scripts/generate_chat_template.py`: CLI for generating and saving chat templates\n\n## Experimental\n- `src/mistral_common/experimental/`: Experimental features\n  - `utils.py`: Utility functions\n  - `tools.py`: Tool calls parser\n  - `think.py`: Thinking parser\n  - `app/`: FastAPI application\n    - `routers.py`: API routers\n    - `main.py`: Application entry point\n    - `models.py`: Pydantic models\n\n## Data\n- `src/mistral_common/data/`: Data files for tokenizers\n\n### Other files and directories\n- `tests/`: Test suite\n- `docs/`: Documentation\n- `.github/workflows/`: CI/CD workflows\n- `.pre-commit-config.yaml`: Pre-commit hooks\n- `pyproject.toml`: Project configuration\n\n## Code Style Guidelines\n\n### Style\n- Respect ruff and mypy rules\n- Naming: snake_case for functions/variables, PascalCase for classes\n- Use Python functionalities supported by Python 3.10\n- Call function arguments explicitly by keyword, not implicitly by position (e.g. `fn(x=1, y=2)`, not `fn(1, 2)`)\n\n### Comments\n- Do NOT write comments that paraphrase or restate what the code already says.\n- Comments should explain the \"why\" (intent, rationale, non-obvious constraints), not the \"what\".\n\n### Imports\n- Use absolute imports for modules within the project\n- Do NOT use wildcard imports\n- Do NOT add import inside `__init__`\n- Use `TYPE_CHECKING` blocks for type-only imports\n- Do NOT use `from __future__ import annotations`\n\n### Type Hints (Required)\n- Use Python's type hints extensively\n- Use modern (Python 3.10+) typing module types\n\n### Error Handling\n- Use custom exceptions from `mistral_common.exceptions`\n- Provide meaningful error messages\n\n### Docstrings (Required)\n- Use Google-style docstrings\n- One-liner for simple functions\n- Multi-line with Args/Returns for complex ones\n- Use `r\"\"\"` for raw docstrings\n- Document all parameters and return values.\n  - Do NOT put types in the docstring for parameters and return values.\n  - For the returns sections, only describe the returned value and do not write its name\n- Include examples where appropriate that can be tested via `doctest`\n\n### Coverage\n- Aim for high, meaningful coverage. Prioritise tests that verify behaviour over hitting a line-count target.\n- New and changed code should be covered by tests.\n- Avoid coverage-only comments. Prefer restructuring so branches are genuinely reachable and tested (e.g. validate inputs and test the error path) over excluding lines.\n\n### Test organization\n\n- New unit tests belong under `tests/unit/` and mirror the source package structure.\n- New integration tests belong under `tests/integration/` and exercise public workflows.\n- Existing tests elsewhere are legacy during the ongoing migration and are removed as their replacements land.\n\n## Development Workflow\n\n1. Set up using uv\n\n```bash\nuv sync --frozen --all-extras --group dev --python 3.12\nsource .venv/bin/activate\nuv run pre-commit install\n```\n\n2.  Make Changes\n- Follow code style guidelines\n- Write tests for new functionality\n- Update documentation\n- When adding dependencies, modify root `pyproject.toml`, then run `uv lock` followed by `uv sync --frozen`\n- Backward Compatibility: Don't break existing functionality\n\n3. Run linter, formatter (Ruff), type checker (mypy) and tests (pytest), including doctests\n\n### Build documentation locally\n\nThe documentation is built with [Zensical](https://zensical.org) (a drop-in successor to MkDocs that reads `mkdocs.yml` natively):\n\n```bash\nuv sync --frozen --all-extras --group docs\nuv run python docs/gen_api_pages.py  # generate the code reference pages\nuv run zensical serve                # live preview at http://localhost:8000\n```\n\nUse `uv run zensical build` for a one-shot static build into `site/`.\n\n### Commit\n- After adding your changes before committing ensure pre-commit is installed or run it manually.\n- Use imperative grammar, start with a verb and be concise.\n\n## Additional Resources\n\n- [Pydantic Documentation](https://docs.pydantic.dev/latest/)\n- [Ruff Documentation](https://docs.astral.sh/ruff/)\n- [Mypy Documentation](https://mypy.readthedocs.io/)\n- [Pytest Documentation](https://docs.pytest.org/)\n</yakaai_skills>"}