mistral-common Documentation

Technical Documentation: mistralai/mistral-common

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

What is it?

mistral-common is a set of tools to help you work with Mistral AI models.

We open-source the tokenizers, validation and normalization code that can be used with our models.

This ensures that you can take full advantage of our models for the following features:

tokenization of text, images and tools calls.
validation and normalization of requests, messages, tool calls, and responses. This is built on top of the Pydantic library.

We also version our tokenizers to guarantee backward compatibility for the models that we release.

For who ?

This library is for you if you want to:

use our models in your own application.
build your own models and want to use the same tokenization and validation code as we do.

How to use it ?

You can install the library using pip:

pip install mistral-common

We propose different dependencies to install depending on your needs:

image: to use the image tokenizers.
audio: to use the audio tokenizers.
hf-hub: to download the tokenizers from the Hugging Face Hub.
sentencepiece: to allow the use of SentencePiece tokenizers. This is now optional as we only release Tekken tokenizers for recent models.
[Experimental] server: to use our tokenizers in a server mode.

Each dependency is optional and can be installed separately or all together using the following commands:

pip install "mistral-common[image]"
pip install "mistral-common[audio]"
pip install "mistral-common[hf-hub]"
pip install "mistral-common[sentencepiece]"
pip install "mistral-common[server]"
pip install "mistral-common[image,audio,hf-hub,sentencepiece,server]"

For more information, please refer to the documentation.

How to contribute?

We welcome contributions to this library. Please read our contribution guide first. If you encounter a bug, have difficulties using mistral-common, or want to propose a feature, search the existing issues and open an issue on our GitHub repository before starting work.

If you want to contribute code after the issue discussion, you may fork the repository and open a pull request. We will review changes that are relevant to the library and meet the contribution guidelines.

# Clone the repository
git clone https://github.com/<your_fork_username>/mistral-common.git
cd mistral-common

# Create a virtual environment and install the dependencies
uv venv
source .venv/bin/activate
uv sync --frozen --all-extras --group dev # --group docs if you want to build the documentation

# Install the pre-commit hooks
uv run pre-commit install

License

This library is licensed under the Apache 2.0 License. See the LICENSE file for more information.

You must not use this library or our models in a manner that infringes, misappropriates, or otherwise violates any third party’s rights, including intellectual property rights.

Index (`docs/index.md`)

Introduction

What is it?

mistral-common is a set of tools to help you work with Mistral models.

We open-source the tokenizers, validation and normalization code that can be used with our models.

This ensures that you can take full advantage of our models for the following features:

tokenization of text, images and tools calls.
validation and normalization of requests, messages, tool calls, and responses. This is built on top of the Pydantic library.

We also version our tokenizers to guarantee backward compatibility for the models that we release.

For who ?

This library is for you if you want to:

use our models in your own application.
build your own models and want to use the same tokenization and validation code as we do.

Explore the following sections:

Quickstart for simple code snippets to use the library for different tasks.
Usage section to install and use the library.
[Code Reference][mistral_common.base] section to see the code documentation.

Launch the documentation locally

To launch the documentation locally, simply run the following command at the root of the repository:

mkdocs serve

Usage Index (`docs/usage/index.md`)

Usage

Installation

See installation instructions.

Getting Started

mistral-common is a collection of common utilities for Mistral to construct valid requests and call tokenizers.

To take full advantage of the features of our models, you will need to use text, images, and tools. Building a valid request can be challenging, so we've created a set of tools to help you get started.

See the following sections for more details:

The requests and their different types.
The images.
The tools.
The tokenizers and their different layer of abstractions that we use.

Examples Inference (`docs/examples/inference.md`)

Inference

We have a few examples of how to use the library with our models:

Chat Completion
Fill-in-the-middle (FIM) Completion
Audio Transcription

Chat Completion

Text-only

python

from mistral_common.protocol.instruct.chunk import TextChunk
from mistral_common.protocol.instruct.messages import (
    AssistantMessage,
    UserMessage,
)
from mistral_common.protocol.instruct.request import ChatCompletionRequest
from mistral_common.tokens.tokenizers.base import SpecialTokenPolicy
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer

repo_id = "mistralai/Mistral-Large-Instruct-2411"
tokenizer = MistralTokenizer.from_hf_hub(repo_id)


messages = [
    UserMessage(content="What is the capital of France?"),
    AssistantMessage(content="The capital of France is Paris."),
    UserMessage(content="And the capital of Spain?"),
]

request = ChatCompletionRequest(messages=messages)
tokenized = tokenizer.encode_chat_completion(request)

# pass tokenized.tokens to your favorite model
print(tokenized.tokens)

# print text to visually see tokens
print(tokenizer.decode(tokens=tokenized.tokens, special_token_policy=SpecialTokenPolicy.KEEP))

Image

text

/* Detailed source-code truncated for AI context efficiency. */

Function calling

text

/* Detailed source-code truncated for AI context efficiency. */

Audio

from mistral_common.protocol.instruct.chunk import TextChunk, AudioChunk
from mistral_common.protocol.instruct.messages import UserMessage, AssistantMessage
from mistral_common.protocol.instruct.request import ChatCompletionRequest
from mistral_common.tokens.tokenizers.base import SpecialTokenPolicy
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistral_common.tokens.tokenizers.audio import Audio
from huggingface_hub import hf_hub_download

repo_id = "mistralai/Voxtral-Mini-3B-2507"
tokenizer = MistralTokenizer.from_hf_hub(repo_id)

obama_file = hf_hub_download("patrickvonplaten/audio_samples", "obama.mp3", repo_type="dataset")
bcn_file = hf_hub_download("patrickvonplaten/audio_samples", "bcn_weather.mp3", repo_type="dataset")

def file_to_chunk(file: str) -> AudioChunk:
    audio = Audio.from_file(file, strict=False)
    return AudioChunk.from_audio(audio)

text_chunk = TextChunk(text="Which speaker do you prefer between the two? Why? How are they different from each other?")
user_msg = UserMessage(content=[file_to_chunk(obama_file), file_to_chunk(bcn_file), text_chunk]).to_openai()


request = ChatCompletionRequest(messages=[user_msg])
tokenized = tokenizer.encode_chat_completion(request)

# pass tokenized.tokens to your favorite audio model
print(tokenized.tokens)
print(tokenized.audios)

# print text to visually see tokens
print(tokenizer.decode(tokens=tokenized.tokens, special_token_policy=SpecialTokenPolicy.KEEP))

FIM

python

from mistral_common.tokens.tokenizers.base import SpecialTokenPolicy
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistral_common.protocol.fim.request import FIMRequest

tokenizer = MistralTokenizer.from_hf_hub("mistralai/Codestral-22B-v0.1")

prefix = """def add("""
suffix = """    return sum"""

request = FIMRequest(prompt=prefix, suffix=suffix)

tokenized = tokenizer.encode_fim(request)

# pass tokenized.tokens to your favorite model
print(tokenized.tokens)

# print text to visually see tokens
print(tokenizer.decode(tokens=tokenized.tokens, special_token_policy=SpecialTokenPolicy.KEEP))

Audio Transcription

python

from mistral_common.protocol.transcription.request import TranscriptionRequest
from mistral_common.tokens.tokenizers.audio import Audio
from mistral_common.tokens.tokenizers.base import SpecialTokenPolicy
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer

from huggingface_hub import hf_hub_download

repo_id = "mistralai/Voxtral-Mini-3B-2507"
tokenizer = MistralTokenizer.from_hf_hub(repo_id)

obama_file = hf_hub_download("patrickvonplaten/audio_samples", "obama.mp3", repo_type="dataset")
audio = Audio.from_file(obama_file, strict=False)

request = TranscriptionRequest(model=repo_id, audio=audio.to_base64(audio.format), language="en")

tokenized = tokenizer.encode_transcription(request)

# pass tokenized.tokens to your favorite audio model
print(tokenized.tokens)
print(tokenized.audios)

# print text to visually see tokens
print(tokenizer.decode(tokens=tokenized.tokens, special_token_policy=SpecialTokenPolicy.KEEP))

Usage Experimental (`docs/usage/experimental.md`)

Mistral-common Experimental API

Context

The experimental module provides access to a FastAPI server designed to handle tokenization and detokenization operations through a REST API. This server features:

Tokenization: Converting chat completion requests into token sequences
Detokenization: Converting token sequences back into human-readable formats to an [AssistantMessage][mistral_common.protocol.instruct.messages.AssistantMessage] object or a raw string.
Chat Completion: Generating an [AssistantMessage][mistral_common.protocol.instruct.messages.AssistantMessage] from a [ChatCompletionRequest][mistral_common.protocol.instruct.request.ChatCompletionRequest] using an engine backend.

This API serves as a bridge between different providers and tokenization needs regardless of the provider programming language.

Features

Tokenizer Version Support: Handles all our tokenizer versions automatically
Tool Call Parsing: Automatically extracts tool calls from token sequences
Think Chunk Support: Handles think chunks for v13+ tokenizers
OpenAI Compatibility: Accepts OpenAI-formatted requests for easier integration

Error Handling

The API provides detailed error messages for:

Invalid token sequences
Empty token lists
Missing tokenizer configuration
Validation errors in requests
Tool call parsing failures

Usage

Installation

To use the experimental API server, you need to install mistral-common with the appropriate dependencies:

bash

pip install mistral-common[server]

Launching the Server

You can launch the server using the following CLI command:

bash

mistral_common serve mistralai/Magistral-Small-2507 [validation_mode] \
--host 127.0.0.1 --port 8000 \
--engine-url http://127.0.0.1:8080 --engine-backend llama_cpp \
--timeout 60

Command Line Options

tokenizer_path: Path to the tokenizer. Can be a HuggingFace model ID or a local path.
validation_mode: Validation mode to use, choices in: "test", "finetuning", "serving", "agnostic" (Optional, defaults to "test")
--host: API host (default: 127.0.0.1)
--port: API port (default: 0 - auto-selects available port)
--engine-url: URL of the engine (default: http://127.0.0.1:8080)
--engine-backend: Engine backend to use (default: llama_cpp)
--timeout: Timeout for the engine (default: 60)

Available Routes

Documentation

Route: / or /docs
Method: GET
Description: The Swagger documentation

The Swagger UI provides interactive documentation for all available endpoints.

Tokenization

Tokenize Request

Route: /v1/tokenize/
Method: POST
Description: Tokenizes a chat completion request
Request Body:
- Chat completion request in either:
  - Mistral-common format ([ChatCompletionRequest][mistral_common.protocol.instruct.request.ChatCompletionRequest])
  - OpenAI-compatible format ([OpenAIChatCompletionRequest][mistral_common.experimental.app.models.OpenAIChatCompletionRequest]). Streaming is not supported.
Response: List of token IDs

Example requests:

text

/* Detailed source-code truncated for AI context efficiency. */

Detokenization

Detokenize to Assistant Message

Route: /v1/detokenize/
Method: POST
Description: Converts tokens to an assistant message with tool call parsing
Request Body:
- List of token IDs
Response: [AssistantMessage][mistral_common.protocol.instruct.messages.AssistantMessage] object with parsed content and tool calls

Example requests:

python

import requests

response = requests.post(
    "http://localhost:8000/v1/detokenize/",
    json=[1, 3, 22177, 1044, 2606, 1584, 1636, 1063, 4]
)
print(response.json())
# {'role': 'assistant', 'content': [{'type': 'text', 'text': 'Hello, how are you?'}], 'tool_calls': None, 'prefix': True}

# With think chunks
response = requests.post(
    "http://localhost:8000/v1/detokenize/",
    json=[12598, 1639, 3648, 2314, 1494, 1046, 34, 6958, 1584, 1032, 1051, 1576, 1114, 1039, 1294, 51567, 33681, 3082, 35, 19587, 1051, 19587, 2]
)
print(response.json())
# {'role': 'assistant', 'content': [{'type': 'text', 'text': 'Let me think about it.'}, {'type': 'thinking', 'thinking': "There are 3 'r' in strawberry ?", 'closed': True}, {'type': 'text', 'text': '$$3$$'}], 'tool_calls': None, 'prefix': False}

# With tool calls
response = requests.post(
    "http://localhost:8000/v1/detokenize/",
    json=[9, 12296, 1095, 99571, 32, 38985, 1039, 3494, 4550, 1576, 1314, 3416, 33681, 2096, 1576, 27965, 4550, 1576, 1114, 1039, 27024, 2]
)
print(response.json())
# {'role': 'assistant', 'content': None, 'tool_calls': [{'id': 'null', 'type': 'function', 'function': {'name': 'count_letters', 'arguments': "{'word': 'strawberry', 'letter': 'r'}"}}], 'prefix': False}

Detokenize to String

Route: /v1/detokenize/string
Method: POST
Description: Converts tokens to a raw string
Request Body:
- tokens: List of token IDs
- special_token_policy: Policy for handling special tokens. See [SpecialTokenPolicy][mistral_common.tokens.tokenizers.base.SpecialTokenPolicy]
Response: Detokenized string

Example request:

python

import requests

response = requests.post("http://localhost:8000/v1/detokenize/string", json={
    "tokens":[1, 3, 22177, 1044, 2606, 1584, 1636, 1063, 4],
    "special_token_policy": 1,
})
print(response.json())
# <s>[INST]Hello, how are you?[/INST]

Chat Completions

Route: /v1/chat/completions
Method: POST
Description: Generates text from a chat completion request. This endpoint forwards the request to the engine server.
Request Body:
- Chat completion request in either:
  - Mistral-common format ([ChatCompletionRequest][mistral_common.protocol.instruct.request.ChatCompletionRequest])
  - OpenAI-compatible format ([OpenAIChatCompletionRequest][mistral_common.experimental.app.models.OpenAIChatCompletionRequest]). Streaming is not supported.
Response: Chat completion response in OpenAI-compatible format

Example request:

python

import requests

response = requests.post("http://localhost:8000/v1/chat/completions", json={
    "messages": [
        {"role": "user", "content": "Hello, how are you?"}
    ]
})
print(response.json())
# {'role': 'assistant', 'content': 'Hello! I am doing well, thank you for asking.'}

Issues and feedback

If you encounter any issues or have feedback, please open an issue or discussion on the GitHub repository.

Usage Images (`docs/usage/images.md`)

Images

Most of the recently released Mistral models support image inputs. Images are represented as [BaseContentChunk][mistral_common.protocol.instruct.chunk.BaseContentChunk] objects within the messages field of the [ChatCompletionRequest][mistral_common.protocol.instruct.request.ChatCompletionRequest]. Encoding an image via a [ImageEncoder][mistral_common.tokens.tokenizers.image.ImageEncoder] will return:

a sequence of special tokens representing the image.
the image normalized as a numpy array.

Supported image formats

Mistral Image encoders use Pillow to decode images and OpenCV to encode. Hence, the supported formats are the same as Pillow's. The images can be provided as:

an [ImageURLChunk][mistral_common.protocol.instruct.chunk.ImageURLChunk]: a pydantic model containing an image URL from which the image will be downloaded.
an [ImageChunk][mistral_common.protocol.instruct.chunk.ImageChunk]: a pydantic model containing a serialized image that can be either a base64 string or a pillow image.

Use an Image encoder with our tokenizer

Our tokenizers can an [ImageEncoder][mistral_common.tokens.tokenizers.image.ImageEncoder] that is configured with [ImageConfig][mistral_common.tokens.tokenizers.image.ImageConfig].

The attributes of the [ImageConfig][mistral_common.tokens.tokenizers.image.ImageConfig] configure how the images will be patched into tokens:

image_patch_size: the square size of a patch in pixels to form one token. E.g if the image is 224x224 and the patch size is 14, then the image will be divided into 16x16 patches.
max_image_size: the maximum size of the image in pixels. If the image is larger, it will be resized to this size.
spatial_merge_size: the number of patches to merge into one token. This is useful to reduce the number of redundant tokens in the image. E.g if the image is 224x224 and the patch size is 14, then the image will be divided into 16x16 patches. If the spatial merge size is 2, then the image will be divided into 8x8 patches.

python

from mistral_common.protocol.instruct.chunk import ImageURLChunk
from mistral_common.tokens.tokenizers.image import ImageEncoder, ImageConfig, SpecialImageIDs

special_ids = SpecialImageIDs(img=10, img_break=11, img_end=12)  # These are normally automatically set by the tokenizer

config = ImageConfig(image_patch_size=14, max_image_size=224, spatial_merge_size=2)

image = ImageURLChunk(image_url="https://live.staticflickr.com/7250/7534338696_b33e941b7d_b.jpg")

encoder = ImageEncoder(config, special_ids)
encoder(image)

Tokenize an image

Let's load the tekken tokenizer used for Mistral Small 3.1's tokenizer to encode and tokenize an image.

python

from huggingface_hub import hf_hub_download

from mistral_common.protocol.instruct.chunk import ImageURLChunk, TextChunk
from mistral_common.protocol.instruct.messages import UserMessage
from mistral_common.protocol.instruct.request import ChatCompletionRequest
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer

model_id = "mistralai/Mistral-Small-3.1-24B-Instruct-2503"

tokenizer = MistralTokenizer.from_hf_hub(repo_id=model_id, token="your_hf_token")

tokenizer.encode_chat_completion(
    ChatCompletionRequest(
        messages=[
            UserMessage(
                content=[
                    ImageURLChunk(image_url="https://live.staticflickr.com/7250/7534338696_b33e941b7d_b.jpg"),
                    TextChunk(text="What is displayed in this image?"),
                ]
            )
        ],
    )
)
# Tokenized(tokens=[1, 3, 10, 10, ...], prefix_ids=None, images=[array[[(0.95238595, 0.95238795, 0.95224484, ...,)]]], audios=[])

The output contains:

the tokens: identifier used by the model for the text. The image is represented into a sequence of special [IMG] tokens with [IMG_BREAK] at regular intervals and [IMG_END] at the end. An image is a grid and each [IMG] represent a patch, the IMG_BREAK tokens the end of a row. The IMG_END token is used to mark the end of the image. The special tokens of images are not directly used by the model, but replaced by the features of an image encoder.
the prefix_ids: Used for FIM (Fill-In-the-Middle) tasks, here it is None.
the images: the images normalized as a numpy array.
the audios: the audios associated with the tokens, here it is empty since there is no audio in the request.

To visualize the tokens as text, decode them with tokenizer.decode(tokens=tokenized.tokens, special_token_policy=SpecialTokenPolicy.KEEP).

Usage Install (`docs/usage/install.md`)

Install

Pip

You can install the library using pip:

pip install mistral-common

We propose different dependencies to install depending on your needs:

image: to use the image tokenizers.
audio: to use the audio tokenizers.
hf-hub: to download the tokenizers from the Hugging Face Hub.
sentencepiece: to allow the use of SentencePiece tokenizers. This is now optional as we only release Tekken tokenizers for recent models.
guidance: to use the grammar-based guided generation feature.
[Experimental] server: to use our tokenizers in a server mode.

Each dependency is optional and can be installed separately or all together using the following commands:

pip install "mistral-common[image]"
pip install "mistral-common[audio]"
pip install "mistral-common[hf-hub]"
pip install "mistral-common[sentencepiece]"
pip install "mistral-common[guidance]"
pip install "mistral-common[server]"
pip install "mistral-common[image,audio,hf-hub,sentencepiece,guidance,server]"

From source

To build it for source, you can clone the repository and install it using uv or pip. We recommend using uv for faster and more reliable dependency resolution:

git clone https://github.com/mistralai/mistral-common.git
cd mistral-common
uv sync --frozen --extra image # or --all-extras to install all dependencies.

For development, you can install the dev group and/or the docs groups:

uv sync --frozen --all-extras --group dev # and/or --group docs.

Usage Requests (`docs/usage/requests.md`)

Requests

To query an AI assistant like Mistral's LeChat or ChatGPT you need to provide the following:

The history of the conversation between the user, the assistant and the tool calls.
The tools available to the assistant.
The context of the request.

In mistral-common, we currently support the following requests types:

Instruct requests:
- Chat completion requests.
- Fill-In-the-Middle completion.
Transcription requests

Every instruct requests should be encoded with its corresponding encode_function function by the tokenizers.

Chat completion

Chat completion consists in a conversation between a user and an assistant. The assistant can call tools to enrich its response. Some of our tokenizers also support the use of images (see the images)

Every chat completion request are defined via [ChatCompletionRequest][mistral_common.protocol.instruct.request.ChatCompletionRequest].

To perform actual task every requests should follow the following structure:

validate the request via a [MistralRequestValidator][mistral_common.protocol.instruct.validator.MistralRequestValidator]
normalize the requests via the [InstructRequestNormalizer][mistral_common.protocol.instruct.normalize.InstructRequestNormalizer].
encode the request.

Using the [MistralTokenizer.encode_chat_completion][mistral_common.tokens.tokenizers.mistral.MistralTokenizer.encode_chat_completion] method will perform all these steps for you.

Following this design ensures minimizing unexpected behavior from the user.

Conversation

A conversation with a model is a sequence of messages. Each message can be a user message, an assistant message, a tool message or a system message. To ease the creation of these messages, we provide a set of Pydantic classes that you can use to create them:

[UserMessage][mistral_common.protocol.instruct.messages.UserMessage]: a message from the user. Users are the ones that interact with the model.
[AssistantMessage][mistral_common.protocol.instruct.messages.AssistantMessage]: a message from the assistant. The assistant is the model itself.
[ToolMessage][mistral_common.protocol.instruct.messages.ToolMessage]: a message from a tool. Tools are functions that the model can call to get information to answer the user's question.
[SystemMessage][mistral_common.protocol.instruct.messages.SystemMessage]: a message from the system. Also called System Prompt, it is a set of instructions that the model should follow to answer the user's question. This allows you to customize the behavior of the model.

Tools

Tools are functions that the model can call to get information to answer the user's question. See the Tools section for more information.

Example

Here is an example of a request where a user asks for the weather in Paris. The model is also given access to a get_current_weather tool to get the weather:

python

from mistral_common.protocol.instruct.messages import UserMessage
from mistral_common.protocol.instruct.request import ChatCompletionRequest
from mistral_common.protocol.instruct.tool_calls import Function, Tool
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer

request = ChatCompletionRequest(
    tools=[
        Tool(
            function=Function(
                name="get_current_weather",
                description="Get the current weather",
                parameters={
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g. San Francisco, CA",
                        },
                        "format": {
                            "type": "string",
                            "enum": ["celsius", "fahrenheit"],
                            "description": "The temperature unit to use. Infer this from the user's location.",
                        },
                    },
                    "required": ["location", "format"],
                },
            )
        )
    ],
    messages=[
        UserMessage(content="What's the weather like today in Paris"),
    ],
)

tokenizer = MistralTokenizer.v3()
tokenizer.encode_chat_completion(request)

FIM

Fill In the Middle (FIM) is a task where the model is given a prefix and a suffix and is asked to fill in the middle. This is useful for code completion, where the model is given a prefix of code and is asked to complete the code.

A pydantic class [FIMRequest][mistral_common.protocol.fim.request.FIMRequest] is defined to ease the creation of these requests.

python

from mistral_common.protocol.fim.request import FIMRequest
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer

request = FIMRequest(
    prompt="def hello_world():\n    print('Hello, world!')",
    suffix="\n\nhello_world()",
)

tokenizer = MistralTokenizer.v3()
tokenizer.encode_fim(request)

Transcription

Transcription is a task where the model is given an audio file and returns the text corresponding to the speech contained in the audio.

A pydantic class [TranscriptionRequest][mistral_common.protocol.transcription.request.TranscriptionRequest] is defined to ease the creation of these requests.

python

from mistral_common.protocol.transcription.request import TranscriptionRequest
from mistral_common.tokens.tokenizers.audio import Audio
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer

from huggingface_hub import hf_hub_download

repo_id = "mistralai/voxtral-mini"
tokenizer = MistralTokenizer.from_hf_hub(repo_id)

obama_file = hf_hub_download("patrickvonplaten/audio_samples", "obama.mp3", repo_type="dataset")
audio = Audio.from_file(obama_file, strict=False)

request = TranscriptionRequest(model=repo_id, audio=audio.to_base64(audio.format), language="en")

tokenizer.encode_transcription(request)

Usage Tokenizers (`docs/usage/tokenizers.md`)

Tokenizers

How do we perform tokenization ?

Tokenization is the process of converting a sequence of characters into a sequence of tokens. To dive deeper into the tokenization process, you can read the tokenization guide on our website.

Historically, we based our tokenizers on the following libraries:

SentencePiece: we moved from this library to the one below.
tiktoken: we use this library for our tekken tokenizers. It is more efficient for multilingual tokenization.

Tokenizer

For our releases, we provide a tokenizer file that you can use to load a tokenizer. If the tokenizer is based on the tiktoken library, the file is generally named tekken.json. Here is how you could load the Mistral Small 3.1 Instruct's tokenizer:

python

from mistral_common.protocol.instruct.messages import UserMessage
from mistral_common.protocol.instruct.request import ChatCompletionRequest
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer

tokenizer = MistralTokenizer.from_hf_hub("mistralai/Mistral-Small-3.1-24B-Instruct-2503", token="your_hf_token")

chat_completion = ChatCompletionRequest(
    messages=[
        UserMessage(role="user", content="Hello, how are you?"),
    ]
)
tokenizer.encode_chat_completion(chat_completion).tokens
# [1, 3, 22177, 1044, 2606, 1584, 1636, 1063, 4]

We provide three layer of abstraction for our tokenizers:

[Tekkenizer][mistral_common.tokens.tokenizers.tekken.Tekkenizer] or [SentencePieceTokenizer][mistral_common.tokens.tokenizers.sentencepiece.SentencePieceTokenizer]: raw tokenizers to handle raw data to convert them into ids and vice versa.
[InstructTokenizer][mistral_common.tokens.tokenizers.base.InstructTokenizer]: instruct tokenizers that wrap the raw tokenizers to add several helper methods for the different tasks (chat completion or FIM). They are versioned.
[MistralTokenizer][mistral_common.tokens.tokenizers.mistral.MistralTokenizer]: mistral tokenizers that validate the requests, see requests section, and call the instruct tokenizers.

For instance, you can directly load the [Tekkenizer][mistral_common.tokens.tokenizers.tekken.Tekkenizer]:

python

from huggingface_hub import hf_hub_download

from mistral_common.tokens.tokenizers.tekken  import Tekkenizer

model_id = "mistralai/Devstral-Small-2505"
tekken_file = hf_hub_download(repo_id=model_id, filename="tekken.json", token="your_hf_token")

tokenizer = Tekkenizer.from_file(tekken_file)
tokenizer.encode("Plz tekkenize me", bos=True, eos=True)
# [1, 4444, 1122, 16058, 3569, 2033, 1639, 2]

See the Examples section for examples on how to use the tokenizers with our models.

Special tokens

Special tokens are tokens that have a special meaning for the model. They are used to mark the beginning or end of a sequence, an image, tools, etc. For example, in the Mistral Small 3.1 Instruct tokenizer, some special tokens include:

<s>: Beginning of a sequence
</s>: End of a sequence
[INST]: Beginning of an instruction
[/INST]: End of an instruction
[TOOL_CALLS]: Beginning of a tool call
[IMG]: Content of an image
[IMG_BREAK]: End of a row in an image
[IMG_END]: End of an image
...

These tokens are defined in the tokenizer configuration file (recommended) or at instantiation for the [Tekkenizer][mistral_common.tokens.tokenizers.tekken.Tekkenizer] (deprecated).

In mistral-common, special tokens are never encoded directly. This means that:

python

tokenizer.encode("<s>")

will not return the ID of the <s> token. Instead, it will return a list of IDs corresponding to the tokenization of the string "<s>". The special token IDs are added directly to the sequence of IDs when encoding the requests.

To add new tokens to the tokenizer and use them correctly, you need to:

Add the tokens to the tokenizer configuration file.
Include the special token IDs in the input IDs when encoding requests.

We are open to suggestions for improvement. Please open an issue if you have any feedback.

Usage Tools (`docs/usage/tools.md`)

Tools

To execute external tools, like searching the web, LLMs need to be given the ability to call such tools whenever they see fit. This is usually achieved by adding all available tools that the LLM can call into the context of the [ChatCompletionRequest][mistral_common.protocol.instruct.request.ChatCompletionRequest] and representing tool calls as well as tool message as its own objects.

Define tools

To define tools in mistral-common, you use the [Tool][mistral_common.protocol.instruct.tool_calls.Tool] class. Let's define a simple tool that fetches the current weather for a given city.

The example function below is not a real call that calls the weather, but it very well be an call to a weather API. We hardcode random numbers here just for demonstrative purposes.

python

from mistral_common.protocol.instruct.tool_calls import Tool, Function


def get_current_weather(location: str, format: str) -> int:
    """Get the current weather in a given location"""
    if "tokyo" in location.lower():
        return 22 if format == "celsius" else 72
    elif "san francisco" in location.lower():
        return 14 if format == "celsius" else 57
    elif "paris" in location.lower():
        return 18 if format == "celsius" else 64
    else:
        return 20 if format == "celsius" else 68

weather_tool = Tool(
    function=Function(
        name="get_current_weather",
        description="Get the current weather",
        parameters={
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA",
                },
                "format": {
                    "type": "string",
                    "enum": ["celsius", "fahrenheit"],
                    "description": "The temperature unit to use. Infer this from the user's location.",
                },
            },
            "required": ["location", "format"],
        },
    )
)

Tool calling and messages

When a model decides to call a tool, it will add a [ToolCall][mistral_common.protocol.instruct.tool_calls.ToolCall] to the [AssistantMessage][mistral_common.protocol.instruct.messages.AssistantMessage]. The tool call will have an id and a [FunctionCall][mistral_common.protocol.instruct.tool_calls.FunctionCall] with the name of the function and the arguments to pass to it.

python

from mistral_common.protocol.instruct.messages import AssistantMessage
from mistral_common.protocol.instruct.request import ChatCompletionRequest
from mistral_common.protocol.instruct.tool_calls import (
    Function,
    FunctionCall,
    Tool,
    ToolCall,
)


AssistantMessage(
    content=None,
    tool_calls=[
        ToolCall(
            id="VvvODy9mT",
            function=FunctionCall(
                name="get_current_weather",
                arguments='{"location": "Paris, France", "format": "celsius"}',
            ),
        )
    ],
)

Then you can execute the function and return the result in a [ToolMessage][mistral_common.protocol.instruct.messages.ToolMessage]. The [ToolMessage][mistral_common.protocol.instruct.messages.ToolMessage] must have the same id as the [ToolCall][mistral_common.protocol.instruct.tool_calls.ToolCall] it's responding to.

python

from mistral_common.protocol.instruct.messages import ToolMessage


ToolMessage(tool_call_id="VvvODy9mT", name="get_current_weather", content="22")

Putting it all together

Now we can put it all together and create a [ChatCompletionRequest][mistral_common.protocol.instruct.request.ChatCompletionRequest] with the tools and the messages. We can then use the [MistralTokenizer][mistral_common.tokens.tokenizers.mistral.MistralTokenizer] to encode the request.

text

/* Detailed source-code truncated for AI context efficiency. */

Now the model can use the tool result to answer the user's question.

--- METRICS ---

Files Extracted: 10
Estimated Token Budget: ~9448 tokens
Canonical Reference: https://codewiki.google/github.com/mistralai/mistral-common

Technical Documentation: mistralai/mistral-common

Overview (`README.md`) Copy Section

What is it? Copy Chapter

For who ? Copy Chapter

How to use it ? Copy Chapter

How to contribute? Copy Chapter

License Copy Chapter

Index (`docs/index.md`) Copy Section

Introduction

What is it? Copy Chapter

For who ? Copy Chapter

Table of contents Copy Chapter

Launch the documentation locally Copy Chapter

Usage Index (`docs/usage/index.md`) Copy Section

Usage

Installation Copy Chapter

Getting Started Copy Chapter

Examples Inference (`docs/examples/inference.md`) Copy Section

Inference

Chat Completion Copy Chapter

Text-only Copy Section

Image Copy Section

Function calling Copy Section

Audio Copy Section

FIM Copy Chapter

Audio Transcription Copy Chapter

Usage Experimental (`docs/usage/experimental.md`) Copy Section

Mistral-common Experimental API

Context Copy Chapter

Features Copy Chapter

Error Handling Copy Chapter

Usage Copy Chapter

Installation Copy Section

Launching the Server Copy Section

Command Line Options

Available Routes Copy Chapter

Documentation Copy Section

Tokenization Copy Section

Tokenize Request

Detokenization Copy Section

Detokenize to Assistant Message

Detokenize to String

Chat Completions Copy Section

Issues and feedback Copy Chapter

Usage Images (`docs/usage/images.md`) Copy Section

Images

Supported image formats Copy Chapter

Use an Image encoder with our tokenizer Copy Chapter

Tokenize an image Copy Chapter

Usage Install (`docs/usage/install.md`) Copy Section

Install

Pip Copy Chapter

From source Copy Chapter

Usage Requests (`docs/usage/requests.md`) Copy Section

Requests

Chat completion Copy Chapter

Conversation Copy Section

Tools Copy Section

Example Copy Section

FIM Copy Chapter

Transcription Copy Chapter

Usage Tokenizers (`docs/usage/tokenizers.md`) Copy Section

Tokenizers

How do we perform tokenization ? Copy Chapter

Tokenizer Copy Chapter

Special tokens Copy Chapter

Usage Tools (`docs/usage/tools.md`) Copy Section

Tools

Define tools Copy Chapter

Tool calling and messages Copy Chapter

Putting it all together Copy Chapter

Explore Related Technologies & Topics

Overview (`README.md`)

What is it?

For who ?

How to use it ?

How to contribute?

License

Index (`docs/index.md`)

What is it?

For who ?

Table of contents

Launch the documentation locally

Usage Index (`docs/usage/index.md`)

Installation

Getting Started

Examples Inference (`docs/examples/inference.md`)

Chat Completion

Text-only

Image

Function calling

Audio

FIM

Audio Transcription

Usage Experimental (`docs/usage/experimental.md`)

Context

Features

Error Handling

Usage

Installation

Launching the Server

Available Routes

Documentation

Tokenization

Detokenization

Chat Completions

Issues and feedback

Usage Images (`docs/usage/images.md`)

Supported image formats

Use an Image encoder with our tokenizer

Tokenize an image

Usage Install (`docs/usage/install.md`)

Pip

From source

Usage Requests (`docs/usage/requests.md`)

Chat completion

Conversation

Tools

Example

FIM

Transcription

Usage Tokenizers (`docs/usage/tokenizers.md`)

How do we perform tokenization ?

Tokenizer

Special tokens

Usage Tools (`docs/usage/tools.md`)

Define tools

Tool calling and messages

Putting it all together