Source/En/Amd Gpu Copy Section

Using TEI on AMD Instinct GPUs (ROCm)

Text Embeddings Inference can run on AMD Instinct GPUs using ROCm. The implementation uses PyTorch's built-in scaled_dot_product_attention as the attention backend.

Prerequisites Copy Chapter

- AMD Instinct GPU (MI200, MI300 series) with ROCm 6.x drivers on the host
- Either a working ROCm PyTorch installation, or the rocm/pytorch:latest Docker image (recommended)

---

The recommended way to get started is to use AMD's official ROCm PyTorch image, which ships with PyTorch and ROCm pre-installed. Alternatively, you can install ROCm PyTorch directly on the host with pip install torch --index-url https://download.pytorch.org/whl/rocm6.2 and skip Step 1.

Step 1: Start the container Copy Chapter

docker run -it --device=/dev/kfd --device=/dev/dri \
  --group-add video --shm-size 8g \
  -v $PWD:/workspace \
  rocm/pytorch:latest bash

Inside the container, clone the TEI repository (or mount it via -v) and run the remaining steps from the repo root.

Step 2: Install Rust Copy Chapter

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source "$HOME/.cargo/env"

Step 3: Install Python dependencies Copy Chapter

PyTorch is already provided by the container image, so install the remaining dependencies without pulling a new torch:

pip install --no-deps -r backends/python/server/requirements-amd.txt
pip install safetensors opentelemetry-api opentelemetry-sdk \
    opentelemetry-exporter-otlp-proto-grpc grpcio-reflection \
    grpc-interceptor einops packaging

Step 4: Generate protobuf stubs Copy Chapter

pip install grpcio-tools==1.62.2 mypy-protobuf==3.6.0 types-protobuf
mkdir -p backends/python/server/text_embeddings_server/pb
python -m grpc_tools.protoc \
    -I backends/proto \
    --python_out=backends/python/server/text_embeddings_server/pb \
    --grpc_python_out=backends/python/server/text_embeddings_server/pb \
    --mypy_out=backends/python/server/text_embeddings_server/pb \
    backends/proto/embed.proto
Fix relative imports in generated files

find backends/python/server/text_embeddings_server/pb/ -name "*.py" \
    -exec sed -i 's/^\(import.*pb2\)/from . \1/g' {} \;
touch backends/python/server/text_embeddings_server/pb/__init__.py

Step 5: Install the Python server package Copy Chapter

pip install -e backends/python/server

Step 6: Build the Rust router Copy Chapter

cargo build --release \
    --no-default-features \
    --features python,http \
    --bin text-embeddings-router

Step 7: Launch TEI Copy Chapter

model=BAAI/bge-base-en-v1.5
./target/release/text-embeddings-router --model-id $model --dtype bfloat16 --port 8080

Once the server is ready, you can test it with a simple embed request:

curl http://localhost:8080/embed \
    -X POST \
    -H 'Content-Type: application/json' \
    -d '{"inputs": "What is Deep Learning?"}'

Verifying GPU detection Copy Chapter

After launch you should see a log line confirming ROCm was detected:

INFO text_embeddings_server::utils::device: ROCm / HIP version: X.Y.Z

You can also verify from Python:

import torch
print(torch.cuda.is_available())  # True
print(torch.version.hip)          # e.g. 6.2.12345-...

Notes Copy Chapter

This is a work in progress — more model support and optimized operations for AMD GPUs are coming soon.

---

Source/En/Cli Arguments Copy Section

CLI arguments

To see all options to serve your models, run the following:

/ Detailed source-code truncated for AI context efficiency. /

---

Source/En/Quick Tour Copy Section

Quick Tour

Set up Copy Chapter

The easiest way to get started with TEI is to use one of the official Docker containers
(see Supported models and hardware to choose the right container).

Hence one needs to install Docker following their installation instructions.

TEI supports inference both on GPU and CPU. If you plan on using a GPU, make sure to check that your hardware is supported by checking this table.
Next, install the NVIDIA Container Toolkit. NVIDIA drivers on your device need to be compatible with CUDA version 12.2 or higher.

Deploy Copy Chapter

Next it's time to deploy your model. Let's say you want to use Qwen/Qwen3-Embedding-0.6B. Here's how you can do this:

model=Qwen/Qwen3-Embedding-0.6B
volume=$PWD/data
docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id $model

<Tip>

We also recommend sharing a volume with the Docker container (volume=$PWD/data) to avoid downloading weights every run.

</Tip>

Inference Copy Chapter

Inference can be performed in 3 ways: using cURL, or via the InferenceClient or OpenAI Python SDKs.

#### cURL

To send a POST request to the TEI endpoint using cURL, you can run the following command:

curl 127.0.0.1:8080/embed \
    -X POST \
    -d '{"inputs":"What is Deep Learning?"}' \
    -H 'Content-Type: application/json'

#### Python

To run inference using Python, you can either use the huggingface_hub Python SDK (recommended) or the openai Python SDK.

##### huggingface_hub

You can install it via pip as pip install --upgrade --quiet huggingface_hub, and then run:

from huggingface_hub import InferenceClient
client = InferenceClient()
embedding = client.feature_extraction("What is deep learning?",
                                      model="http://localhost:8080/embed")
print(len(embedding[0]))

#### OpenAI
To send requests to the OpenAI Embeddings API exposed on Text Embeddings Inference (TEI) with the OpenAI Python SDK, you can install it as pip install --upgrade openai, and then run the following snippet:

import os
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key= "-")
response = client.embeddings.create(
    model="text-embeddings-inference",
    input="What is Deep Learning?",
)
print(response.data[0].embedding)

Alternatively, you can also send the request with cURL as follows:

curl http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "input": "What is Deep Learning?",
    "model": "text-embeddings-inference",
    "encoding_format": "float"
  }'

Re-rankers and sequence classification Copy Chapter

TEI also supports re-ranker and classic sequence classification models.

Re-rankers Copy Section

Rerankers, also called cross-encoders, are sequence classification models with a single class that score the similarity between a query and a text. See this blogpost by
the LlamaIndex team to understand how you can use re-rankers models in your RAG pipeline to improve
downstream performance.

Let's say you want to use BAAI/bge-reranker-large. First, you can deploy it like so:

model=BAAI/bge-reranker-large
volume=$PWD/data
docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id $model

Once you have deployed a model, you can use the rerank endpoint to rank the similarity between a query and a list of texts. With cURL this can be done like so:

curl 127.0.0.1:8080/rerank \
    -X POST \
    -d '{"query":"What is Deep Learning?", "texts": ["Deep Learning is not...", "Deep learning is..."], "raw_scores": false}' \
    -H 'Content-Type: application/json'

Sequence classification models Copy Section

You can also use classic Sequence Classification models like SamLowe/roberta-base-go_emotions:

model=SamLowe/roberta-base-go_emotions
volume=$PWD/data
docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id $model

Once you have deployed the model you can use the predict endpoint to get the emotions most associated with an input:

curl 127.0.0.1:8080/predict \
    -X POST \
    -d '{"inputs":"I like you."}' \
    -H 'Content-Type: application/json'

Batching Copy Chapter

You can send multiple inputs in a batch. For example, for embeddings:

curl 127.0.0.1:8080/embed \
    -X POST \
    -d '{"inputs":["Today is a nice day", "I like you"]}' \
    -H 'Content-Type: application/json'

And for Sequence Classification:

curl 127.0.0.1:8080/predict \
    -X POST \
    -d '{"inputs":[["I like you."], ["I hate pineapples"]]}' \
    -H 'Content-Type: application/json'

Air gapped deployment Copy Chapter

To deploy Text Embeddings Inference in an air-gapped environment, first download the weights and then mount them inside
the container using a volume.

For example:

(Optional) create a models directory

mkdir models
cd models
Make sure you have git-lfs installed (https://git-lfs.com)

git lfs install
git clone https://huggingface.co/Alibaba-NLP/gte-base-en-v1.5
Set the models directory as the volume path

volume=$PWD
Mount the models directory inside the container with a volume and set the model ID

docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id /data/gte-base-en-v1.5

---

CONTRIBUTING Copy Section

Contribute to Text Embeddings Inference (TEI)

Everyone is welcome to contribute, and we value everybody's contribution. Code
contributions are not the only way to help the community. Answering questions, helping
others, and improving the documentation are also immensely valuable.

It also helps us if you spread the word! Reference the library in blog posts
about the awesome projects it made possible, shout out on Twitter every time it has
helped you, or simply ⭐️ the repository to say thank you.

However you choose to contribute, please be mindful and respect our
code of conduct.

This guide was heavily inspired by the awesome scikit-learn guide to contributing.

Ways to contribute Copy Chapter

There are several ways you can contribute to Text Embeddings Inference (TEI).

* Fix outstanding issues with the existing code.
* Submit issues related to bugs or desired new features.
* Contribute to the examples or to the documentation.

All contributions are equally valuable to the community. 🥰

Fixing outstanding issues Copy Chapter

If you notice an issue with the existing code and have a fix in mind, feel free to start contributing and open
a Pull Request!

Submitting a bug-related issue or feature request Copy Chapter

Do your best to follow these guidelines when submitting a bug-related issue or a feature
request. It will make it easier for us to come back to you quickly and with good
feedback.

Did you find a bug? Copy Section

The Text Embeddings Inference (TEI) solution is robust and reliable thanks to users who report the problems they encounter.

Before you report an issue, we would really appreciate it if you could make sure the bug was not
already reported (use the search bar on GitHub under Issues). Your issue should also be related to bugs in the
library itself, and not your code.

Once you've confirmed the bug hasn't already been reported, please include the following information in your issue so
we can quickly resolve it:

* Your OS type and version, as well as your environment versions (versions of rust, python, and dependencies).
* A short, self-contained, code snippet that allows us to reproduce the bug.
The full* traceback if an exception is raised.
* Attach any other additional information, like screenshots, you think may help.

Do you want a new feature? Copy Section

If there is a new feature you'd like to see in Text Embeddings Inference (TEI), please open an issue and describe:

1. What is the motivation behind this feature? Is it related to a problem or frustration with the library? Is it
a feature related to something you need for a project? Is it something you worked on and think it could benefit
the community?

Whatever it is, we'd love to hear about it!

2. Describe your requested feature in as much detail as possible. The more you can tell us about it, the better
we'll be able to help you.
3. Provide a code snippet that demonstrates the feature's usage.
4. If the feature is related to a paper, please include a link.

If your issue is well written we're already 80% of the way there by the time you create it.

We have added templates
to help you get started with your issue.

Do you want to implement a new model? Copy Chapter

New models are constantly released and if you want to implement a new model, please provide the following information:

* A short description of the model and a link to the paper.
* Link to the implementation if it is open-sourced.
* Link to the model weights if they are available.

If you are willing to contribute the model yourself, let us know so we can help you add it to Text Embeddings Inference (TEI)!

Do you want to add documentation? Copy Chapter

We're always looking for improvements to the documentation that make it more clear and accurate. Please let us know
how the documentation can be improved such as typos and any content that is missing, unclear or inaccurate. We'll be
happy to make the changes or help you make a contribution if you're interested!

I want to become a maintainer of the project. How do I get there? Copy Chapter

Text Embeddings Inference (TEI) is a project led and managed by Hugging Face as it powers our internal services. However, we are happy to have
motivated individuals from other organizations join us as maintainers with the goal of making TEI the best inference
service for embedding models on production on production.

If you are such an individual (or organization), please reach out to us and let's collaborate.

---

README Copy Section

<div align="center">

Text Embeddings Inference

<a href="https://github.com/huggingface/text-embeddings-inference">
<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/huggingface/text-embeddings-inference?style=social">
</a>
<a href="https://huggingface.github.io/text-embeddings-inference">
<img alt="Swagger API documentation" src="https://img.shields.io/badge/API-Swagger-informational">
</a>

A blazing fast inference solution for text embeddings models.

Benchmark for BAAI/bge-base-en-v1.5 on an NVIDIA A10 with a sequence
length of 512 tokens:

<p>
<img src="assets/bs1-lat.png" width="400" />
<img src="assets/bs1-tp.png" width="400" />
</p>
<p>
<img src="assets/bs32-lat.png" width="400" />
<img src="assets/bs32-tp.png" width="400" />
</p>

</div>

Table of contents Copy Chapter

- Get Started
- Supported Models
- Docker
- Docker Images
- API Documentation
- Using a private or gated model
- Air gapped deployment
- Using Re-rankers models
- Using Sequence Classification models
- Using SPLADE pooling
- Distributed Tracing
- gRPC
- Local Install
- Apple Silicon (Homebrew)
- Docker Build
- ARM64 / aarch64
- AMD Instinct GPUs (ROCm)
- Examples

Text Embeddings Inference (TEI) is a toolkit for deploying and serving open source text embeddings and sequence
classification models. TEI enables high-performance extraction for the most popular models, including FlagEmbedding,
Ember, GTE and E5. TEI implements many features such as:

* No model graph compilation step
* Metal support for local execution on Macs
* Small docker images and fast boot times. Get ready for true serverless!
* Token based dynamic batching
* Optimized transformers code for inference using Flash Attention,
Candle
and cuBLASLt
* Safetensors weight loading
* ONNX weight loading
* Production ready (distributed tracing with Open Telemetry, Prometheus metrics)

Get Started Copy Chapter

Supported Models Copy Section

#### Text Embeddings

Text Embeddings Inference currently supports Nomic, BERT, CamemBERT, XLM-RoBERTa models with absolute positions, JinaBERT
model with Alibi positions and Mistral, Alibaba GTE, Qwen2 models with Rope positions, MPNet, ModernBERT, Qwen3, and Gemma3.

Below are some examples of the currently supported models:

| MTEB Rank | Model Size | Model Type | Model ID |
|-----------|------------------------|----------------|--------------------------------------------------------------------------------------------------|
| 2 | 7.57B (Very Expensive) | Qwen3 | Qwen/Qwen3-Embedding-8B |
| 3 | 4.02B (Very Expensive) | Qwen3 | Qwen/Qwen3-Embedding-4B |
| 4 | 509M | Qwen3 | Qwen/Qwen3-Embedding-0.6B |
| 6 | 7.61B (Very Expensive) | Qwen2 | Alibaba-NLP/gte-Qwen2-7B-instruct |
| 7 | 560M | XLM-RoBERTa | intfloat/multilingual-e5-large-instruct |
| 8 | 308M | Gemma3 | google/embeddinggemma-300m (gated) |
| 15 | 1.78B (Expensive) | Qwen2 | Alibaba-NLP/gte-Qwen2-1.5B-instruct |
| 18 | 7.11B (Very Expensive) | Mistral | Salesforce/SFR-Embedding-2_R |
| 35 | 568M | XLM-RoBERTa | Snowflake/snowflake-arctic-embed-l-v2.0 |
| 41 | 305M | Alibaba GTE | Snowflake/snowflake-arctic-embed-m-v2.0 |
| 52 | 335M | BERT | WhereIsAI/UAE-Large-V1 |
| 58 | 137M | NomicBERT | nomic-ai/nomic-embed-text-v1 |
| 79 | 137M | NomicBERT | nomic-ai/nomic-embed-text-v1.5 |
| 103 | 109M | MPNet | sentence-transformers/all-mpnet-base-v2 |
| N/A | 475M-A305M | NomicBERT | nomic-ai/nomic-embed-text-v2-moe |
| N/A | 434M | Alibaba GTE | Alibaba-NLP/gte-large-en-v1.5 |
| N/A | 396M | ModernBERT | answerdotai/ModernBERT-large |
| N/A | 340M | Qwen3 | voyageai/voyage-4-nano |
| N/A | 137M | JinaBERT | jinaai/jina-embeddings-v2-base-en |
| N/A | 137M | JinaBERT | jinaai/jina-embeddings-v2-base-code |

To explore the list of best performing text embeddings models, visit the
Massive Text Embedding Benchmark (MTEB) Leaderboard.

#### Sequence Classification and Re-Ranking

Text Embeddings Inference currently supports CamemBERT, and XLM-RoBERTa Sequence Classification models with absolute positions.

Below are some examples of the currently supported models:

| Task | Model Type | Model ID |
|--------------------|-------------|-----------------------------------------------------------------------------------------------------------------|
| Re-Ranking | XLM-RoBERTa | BAAI/bge-reranker-large |
| Re-Ranking | XLM-RoBERTa | BAAI/bge-reranker-base |
| Re-Ranking | GTE | Alibaba-NLP/gte-multilingual-reranker-base |
| Re-Ranking | ModernBert | Alibaba-NLP/gte-reranker-modernbert-base |
| Sentiment Analysis | RoBERTa | SamLowe/roberta-base-go_emotions |

Docker Copy Section

model=Qwen/Qwen3-Embedding-0.6B
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id $model

And then you can make requests like

curl 127.0.0.1:8080/embed \
    -X POST \
    -d '{"inputs":"What is Deep Learning?"}' \
    -H 'Content-Type: application/json'

Note: To use GPUs, you need to install
the NVIDIA Container Toolkit.
NVIDIA drivers on your machine need to be compatible with CUDA version 12.2 or higher.

To see all options to serve your models:

/ Detailed source-code truncated for AI context efficiency. /

Docker Images Copy Section

Text Embeddings Inference ships with multiple Docker images that you can use to target a specific backend:

| Architecture | Platform | Image |
|----------------------------------------|----------|-------------------------------------------------------------------------|
| CPU | x86_64 | ghcr.io/huggingface/text-embeddings-inference:cpu-1.9 |
| CPU | aarch64 | ghcr.io/huggingface/text-embeddings-inference:cpu-arm64-1.9 |
| Volta | x86_64 | NOT SUPPORTED |
| Turing (T4, RTX 2000 series, ...) | x86_64 | ghcr.io/huggingface/text-embeddings-inference:turing-1.9 (experimental) |
| Ampere 8.0 (A100, A30) | x86_64 | ghcr.io/huggingface/text-embeddings-inference:1.9 |
| Ampere 8.6 (A10, A40, ...) | x86_64 | ghcr.io/huggingface/text-embeddings-inference:86-1.9 |
| Ada Lovelace (RTX 4000 series, ...) | x86_64 | ghcr.io/huggingface/text-embeddings-inference:89-1.9 |
| Hopper (H100) | x86_64 | ghcr.io/huggingface/text-embeddings-inference:hopper-1.9 |
| Blackwell 10.0 (B200, GB200, ...) | x86_64 | ghcr.io/huggingface/text-embeddings-inference:100-1.9 (experimental) |
| Blackwell 12.0 (GeForce RTX 50X0, ...) | x86_64 | ghcr.io/huggingface/text-embeddings-inference:120-1.9 (experimental) |
| Blackwell 12.1 (DGX Spark GB10, ...) | multi | ghcr.io/huggingface/text-embeddings-inference:121-1.9 (experimental) |

Warning: Flash Attention is turned off by default for the Turing image as it suffers from precision issues.
You can turn Flash Attention v1 ON by using the USE_FLASH_ATTENTION=True environment variable.

API documentation Copy Section

You can consult the OpenAPI documentation of the text-embeddings-inference REST API using the /docs route.
The Swagger UI is also available
at: https://huggingface.github.io/text-embeddings-inference.

Using a private or gated model Copy Section

You have the option to utilize the HF_TOKEN environment variable for configuring the token employed by
text-embeddings-inference. This allows you to gain access to protected resources.

For example:

1. Go to https://huggingface.co/settings/tokens
2. Copy your CLI READ token
3. Export HF_TOKEN=<your CLI READ token>

or with Docker:

model=<your private model>
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
token=<your CLI READ token>
docker run --gpus all -e HF_TOKEN=$token -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id $model

Air gapped deployment Copy Section

To deploy Text Embeddings Inference in an air-gapped environment, first download the weights and then mount them inside
the container using a volume.

For example:

(Optional) create a models directory

mkdir models
cd models
Make sure you have git-lfs installed (https://git-lfs.com)

git lfs install
git clone https://huggingface.co/Qwen/Qwen3-Embedding-0.6B
Set the models directory as the volume path

volume=$PWD
Mount the models directory inside the container with a volume and set the model ID

docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id /data/Qwen3-Embedding-0.6B

Using Re-rankers models Copy Section

text-embeddings-inference v0.4.0 added support for CamemBERT, RoBERTa, XLM-RoBERTa, and GTE Sequence Classification models.
Re-rankers models are Sequence Classification cross-encoders models with a single class that scores the similarity
between a query and a text.

See this blogpost by
the LlamaIndex team to understand how you can use re-rankers models in your RAG pipeline to improve
downstream performance.

model=BAAI/bge-reranker-large
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id $model

And then you can rank the similarity between a query and a list of texts with:

curl 127.0.0.1:8080/rerank \
    -X POST \
    -d '{"query": "What is Deep Learning?", "texts": ["Deep Learning is not...", "Deep learning is..."]}' \
    -H 'Content-Type: application/json'

Using Sequence Classification models Copy Section

You can also use classic Sequence Classification models like SamLowe/roberta-base-go_emotions:

model=SamLowe/roberta-base-go_emotions
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id $model

Once you have deployed the model you can use the predict endpoint to get the emotions most associated with an input:

curl 127.0.0.1:8080/predict \
    -X POST \
    -d '{"inputs":"I like you."}' \
    -H 'Content-Type: application/json'

Using SPLADE pooling Copy Section

You can choose to activate SPLADE pooling for Bert and Distilbert MaskedLM architectures:

model=naver/efficient-splade-VI-BT-large-query
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id $model --pooling splade

Once you have deployed the model you can use the /embed_sparse endpoint to get the sparse embedding:

curl 127.0.0.1:8080/embed_sparse \
    -X POST \
    -d '{"inputs":"I like you."}' \
    -H 'Content-Type: application/json'

Distributed Tracing Copy Section

text-embeddings-inference is instrumented with distributed tracing using OpenTelemetry. You can use this feature
by setting the address to an OTLP collector with the --otlp-endpoint argument.

gRPC Copy Section

text-embeddings-inference offers a gRPC API as an alternative to the default HTTP API for high performance
deployments. The API protobuf definition can be
found here.

You can use the gRPC API by adding the -grpc tag to any TEI Docker image. For example:

model=Qwen/Qwen3-Embedding-0.6B
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
docker run --gpus all -p 8080:80 -v $volume:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cuda-1.9-grpc --model-id $model

grpcurl -d '{"inputs": "What is Deep Learning"}' -plaintext 0.0.0.0:8080 tei.v1.Embed/Embed

Local install Copy Chapter

Apple Silicon (Homebrew) Copy Section

On Apple Silicon (M1/M2/M3/M4), you can install a prebuilt binary via Homebrew:

brew install text-embeddings-inference

Then launch Text Embeddings Inference with Metal acceleration:

model=Qwen/Qwen3-Embedding-0.6B
text-embeddings-router --model-id $model --port 8080

CPU Copy Section

You can also opt to install text-embeddings-inference locally.

First install Rust:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Then run:

On x86 with ONNX backend (recommended)

cargo install --path router -F ort
On x86 with Intel backend

cargo install --path router -F mkl
On M1 or M2

cargo install --path router -F metal

You can now launch Text Embeddings Inference on CPU with:

model=Qwen/Qwen3-Embedding-0.6B
text-embeddings-router --model-id $model --port 8080

Note: on some machines, you may also need the OpenSSL libraries and gcc. On Linux machines, run:

sudo apt-get install libssl-dev gcc -y

CUDA Copy Section

GPUs with CUDA compute capabilities < 7.5 are not supported (V100, Titan V, GTX 1000 series, ...).

Make sure you have CUDA and the NVIDIA drivers installed. NVIDIA drivers on your device need to be compatible with CUDA
version 12.2 or higher. You also need to add the NVIDIA binaries to your path:

export PATH=$PATH:/usr/local/cuda/bin

Then run the following (might take a while as it needs to compile the CUDA kernels):

On Turing GPUs (T4, RTX 2000 series ... )

cargo install --path router -F candle-cuda-turing
On Ampere, Ada Lovelace, Hopper and Blackwell

cargo install --path router -F candle-cuda

You can now launch Text Embeddings Inference on GPU as follows:

model=Qwen/Qwen3-Embedding-0.6B
text-embeddings-router --model-id $model --port 8080

Docker Copy Chapter

You can build the CPU container with Docker as:

docker build -f Dockerfile .

To build the CUDA containers, you need to know the compute cap of the GPU you will be using
at runtime, to build the image accordingly:

Get submodule dependencies

git submodule update --init
Example for Turing (T4, RTX 2000 series, ...)

runtime_compute_cap=75
Example for Ampere (A100, ...)

runtime_compute_cap=80
Example for Ampere (A10, ...)

runtime_compute_cap=86
Example for Ada Lovelace (RTX 4000 series, ...)

runtime_compute_cap=89
Example for Hopper (H100, ...)

runtime_compute_cap=90
Example for Blackwell (B200, GB200, ...)

runtime_compute_cap=100
Example for Blackwell (GeForce RTX 50X0, RTX PRO 6000, ...)

runtime_compute_cap=120
Example for Blackwell GB10 (DGX Spark)

runtime_compute_cap=121
docker build . -f Dockerfile-cuda --build-arg CUDA_COMPUTE_CAP=$runtime_compute_cap

ARM64 / aarch64 Copy Section

#### CPU-only (Apple Silicon, Ampere, Graviton)

For ARM64 hosts without NVIDIA GPUs, use the CPU Dockerfile. Inference runs on CPU cores
only (no Metal/MPS support via Docker).

docker build . -f Dockerfile-arm64 --platform=linux/arm64

#### CUDA on ARM64 (DGX Spark, Jetson)

For ARM64 hosts with NVIDIA GPUs, build Dockerfile-cuda with the appropriate compute
capability and --platform linux/arm64:

DGX Spark (GB10, sm_121)

docker build . -f Dockerfile-cuda \
  --build-arg CUDA_COMPUTE_CAP=121 \
  --platform linux/arm64
Future ARM64 + Blackwell devices (sm_120)

docker build . -f Dockerfile-cuda \
  --build-arg CUDA_COMPUTE_CAP=120 \
  --platform linux/arm64

AMD Instinct GPUs (ROCm) — experimental Copy Chapter

TEI has experimental support for AMD Instinct GPUs (MI200, MI300 series) via ROCm. You can use the rocm/pytorch:latest Docker image or a bare-metal ROCm installation. TEI will auto-detect the GPU at startup.

For full setup instructions, see the AMD Instinct GPU guide.

Examples Copy Chapter

- Set up an Inference Endpoint with TEI
- RAG containers with TEI

---