inference Documentation

pip install xinference
xinference-local

curl -fsSL https://raw.githubusercontent.com/xorbitsai/inference/main/scripts/install.sh | sh

irm https://raw.githubusercontent.com/xorbitsai/inference/main/scripts/install.ps1 | iex

docker run --name xinference -d -p 9997:9997 -e XINFERENCE_HOME=/data -v </on/your/host>:/data --gpus all xprobe/xinference:latest xinference-local -H 0.0.0.0

# add repo
helm repo add xinference https://xorbitsai.github.io/xinference-helm-charts

# update indexes and query xinference versions
helm repo update xinference
helm search repo xinference/xinference --devel --versions

# install xinference
helm install xinference xinference/xinference -n xinference --version 0.0.1-v<xinference_release_version>

@inproceedings{lu2024xinference,
    title = "Xinference: Making Large Model Serving Easy",
    author = "Lu, Weizheng and Xiong, Lingfeng and Zhang, Feng and Qin, Xuye and Chen, Yueguo",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-demo.30",
    pages = "291--300",
}

python gen_docs.py

from xinference.client import Client

client = Client("http://localhost:9997")
model = client.get_model("MODEL_UID")

# Chat to LLM
model.chat(
   messages=[{"role": "system", "content": "You are a helpful assistant"}, {"role": "user", "content": "What is the largest animal?"}],
   generate_config={"max_tokens": 1024}
)

# Chat to VL model
model.chat(
   messages=[
     {
        "role": "user",
        "content": [
           {"type": "text", "text": "What’s in this image?"},
           {
              "type": "image_url",
              "image_url": {
                 "url": "http://i.epochtimes.com/assets/uploads/2020/07/shutterstock_675595789-600x400.jpg",
              },
           },
        ],
     }
  ],
  generate_config={"max_tokens": 1024}
)

from xinference.client import Client

client = Client("http://localhost:9997")
model = client.get_model("MODEL_UID")

model.create_embedding("What is the capital of China?")

from xinference.client import Client

client = Client("http://localhost:9997")
model = client.get_model("MODEL_UID")

model.text_to_image("An astronaut walking on the mars")

from xinference.client import Client

client = Client("http://localhost:9997")
model = client.get_model("MODEL_UID")

with open("speech.mp3", "rb") as audio_file:
    model.transcriptions(audio_file.read())

from xinference.client import Client

client = Client("http://localhost:9997")
model = client.get_model("MODEL_UID")

query = "A man is eating pasta."
corpus = [
  "A man is eating food.",
  "A man is eating a piece of bread.",
  "The girl is carrying a baby.",
  "A man is riding a horse.",
  "A woman is playing violin."
]
print(model.rerank(corpus, query))

from xinference.client import Client

client = Client("http://localhost:9997")
model = client.get_model("MODEL_UID")

model.text_to_video("")

.. grid-item-card::  Install Xinference
  :link: installation
  :link-type: ref

  Install Xinference on Linux, Windows, and macOS.

.. grid-item-card::  Try it out!
  :link: using_xinference
  :link-type: ref

  Start by running Xinference on a local machine.

Explore a wide range of models supported by Xinference.

Register model weights and turn it into an API.

.. grid-item-card::  Chat & Generate
  :link: chat
  :link-type: ref

  Learn how to chat with LLMs in Xinference.

.. grid-item-card::  Tools
  :link: tools
  :link-type: ref

  Learn how to connect LLM with external tools.

.. grid-item-card::  Embeddings
  :link: embed
  :link-type: ref

  Learn how to create text embeddings in Xinference.

.. grid-item-card::  Rerank
  :link: rerank
  :link-type: ref

  Learn how to use rerank models in Xinference.

.. grid-item-card::  Images
  :link: image
  :link-type: ref

  Learn how to generate images with Xinference.

.. grid-item-card::  Multimodal
  :link: multimodal
  :link-type: ref

  Learn how to process images and audio with LLMs.

Learn how to turn audio into text or text into audio with Xinference.

Learn how to generate video with Xinference.

.. grid-item-card::  World
  :link: world
  :link-type: ref

  Learn how to generate environment videos with world models in Xinference.

.. grid-item-card::  Flexible
  :link: flexible
  :link-type: ref

  Learn how to inference traditional ML models with Xinference.

.. div:: sd-font-weight-normal sd-fs-5
     
     Get Latest News

  .. grid:: 1
     :gutter: 3

     .. grid-item-card::  
        :link: https://twitter.com/Xorbitsio

        :fab:`twitter` Follow us on Twitter

     .. grid-item-card::
        :link: https://zhihu.com/org/xorbits
        :class-item: zh-cn-only

        :fab:`zhihu` Read our blogs

.. div:: sd-font-weight-normal sd-fs-5

     Get Support

  .. grid:: 1
     :gutter: 3

     .. grid-item-card::
        :link: https://discord.gg/Xw9tszSkr5

        :fab:`discord` Find community on Discord

     .. grid-item-card::
        :link: https://t.me/+nCNpwmySwk9iYmI1

        :fab:`telegram` Join our Telegram group

     .. grid-item-card::  
        :link: https://github.com/xorbitsai/inference/issues/new/choose

        :fab:`github` Open an issue

.. div:: sd-fs-5

     Contribute to Xinference

  .. grid:: 1
     :gutter: 3

     .. grid-item-card::  
        :link: https://github.com/xorbitsai/inference/pulls

        :fab:`github` Create a pull request

pip install pre-commit

pre-commit install

pre-commit run --files <files you have modified>

pre-commit run --from-ref=upstream/main --to-ref=HEAD --all-files

You may consider periodically running ``pre-commit gc`` to clean up repos
which are no longer used.

If you have conflicting installations of ``virtualenv``, if could lead to
errors - refer to `here <https://github.com/pypa/virtualenv/issues/1875>`_.

Also, due to a `bug in virtualenv <https://github.com/pypa/virtualenv/issues/1986>`_,
you may run into issues if you're using conda. To solve this, you can downgrade
``virtualenv`` to version ``20.0.33``.

pip install "jsonschema>=4.18,<5"
python xinference/_model_catalog.py validate --all
python xinference/_model_catalog.py validate xinference/model/llm/models --model-type llm

export https_proxy=YourProxyAddress

contributing_environment
contributing_codebase
xinference_internals

Endpoint of Xinference, used to connect to Xinference service.
Default value is http://127.0.0.1:9997 , you can get it through logs.

.. _actor_transport_authentication:

XOSCAR_AUTH_KEY
~~~~~~~~~~~~~~~
Shared secret for Xoscar actor transport authentication, supported by Xoscar
0.12.0 and later. Unset or empty keeps the legacy unauthenticated transport;
upgrading alone does not enable protection. HTTP API authentication does not
protect actor ports. See the `security advisory
<https://github.com/xorbitsai/xoscar/security/advisories/GHSA-mmvj-m246-m9vf>`_.

Generate one random secret per deployment with the following command. Distribute
the same value through your secret-management system before starting
``xinference-local``, ``xinference-supervisor``, every ``xinference-worker``,
and any direct Xoscar actor clients. Child processes, including model virtual
environments, inherit the variable. HTTP clients do not need this secret.

.. code-block:: bash

   python -c "import secrets; print(secrets.token_hex(32))"
   export XOSCAR_AUTH_KEY='<same-random-deployment-secret>'

Upgrade all actor peers to Xoscar 0.12.0 or later before enabling the key, then
coordinate a restart with the same key on every peer. Keyed and unkeyed peers
cannot communicate; there is no automatic downgrade or mixed-mode rolling
upgrade. Key rotation also requires a coordinated restart. For ``docker run``,
pass the exported variable with ``-e XOSCAR_AUTH_KEY``; for Kubernetes, inject
the same Secret into each server container's environment.

Socket and UCX messages are authenticated before deserialization. Anyone holding
the key is a trusted peer with Python execution privileges. Deployment
administrators must restrict actor ports to intended cluster peers. Authentication
does not encrypt traffic or prevent replay; authenticated GPU transfers stage
device buffers in host memory for hashing and add CPU and host-memory overhead.

XINFERENCE_TRUST_REMOTE_CODE

LLM auto-registration reads files only under ``XINFERENCE_MODEL_DIR`` and ``XINFERENCE_CACHE_DIR`` by default. Configure additional directories in the server environment with this variable, separated by ``:`` on Unix or ``;`` on Windows. Referenced files and symbolic-link targets must also lie under an allowed directory. For Hugging Face caches, allow the repository root containing both snapshots and blobs.

XINFERENCE_FLEXIBLE_LAUNCHERS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The shipped Flexible launchers remain available. Permit additional launchers by listing their exact dotted names in this comma-separated variable before starting workers. Allowlisted launchers execute Python code with worker privileges; add only trusted implementations.

XINFERENCE_MODEL_SRC
~~~~~~~~~~~~~~~~~~~~~
Modelhub used for downloading models. Options are "huggingface", "modelscope"
and "auto". When unset or set to "auto", Xinference probes whether the Hugging
Face endpoint (including ``HF_ENDPOINT``) would use an environment-configured
proxy. If so, or if the endpoint is not directly reachable, it falls back to
"modelscope"; otherwise it picks "huggingface". ``NO_PROXY`` is honored. Set an
explicit value to pin the download source. An explicit value can be overridden
for an individual launch with ``--download_hub``.

XINFERENCE_HUB_DETECT_TIMEOUT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Timeout in seconds for probing hub connectivity when the download hub is
automatically detected. Default is 3.

.. _environments_xinference_home:

XINFERENCE_HOME
~~~~~~~~~~~~~~~~
By default, Xinference uses ``<HOME>/.xinference`` as home path to store
necessary files such as logs and models, where ``<HOME>`` is the home
path of current user. You can change this directory by configuring this environment
variable.

XINFERENCE_SYSTEM_SETTINGS_PATH

The maximum number of failed health checks tolerated at Xinference startup.
Default value is 5.

XINFERENCE_HEALTH_CHECK_INTERVAL
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Health check interval (seconds) at Xinference startup.
Default value is 5.

XINFERENCE_HEALTH_CHECK_TIMEOUT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Health check timeout (seconds) at Xinference startup.
Default value is 10.

XINFERENCE_MODEL_GPU_MEMORY_CACHE_TTL
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Maximum age in seconds of the last valid per-model GPU-memory snapshot retained
by the supervisor after a worker omits the field because collection failed or
was skipped. An explicit empty snapshot still clears the cache immediately.
Default value is 90.

XINFERENCE_DISABLE_HEALTH_CHECK
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Xinference will automatically report health check at Xinference startup.
Setting this environment to 1 can disable health check.

XINFERENCE_DISABLE_METRICS
~~~~~~~~~~~~~~~~~~~~~~~~~~
Xinference will by default enable the metrics exporter on the supervisor and worker.
Setting this environment to 1 will disable the /metrics endpoint on the supervisor
and the HTTP service (only provide the /metrics endpoint) on the worker.

XINFERENCE_DOWNLOAD_MAX_ATTEMPTS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Maximum download retry attempts for model files.
Default value is 3.

XINFERENCE_LLMMAN_BIN
~~~~~~~~~~~~~~~~~~~~~
Path to the ``llmman`` executable used to resolve ``oci://`` model URIs.
Defaults to ``llmman`` on ``PATH``. Set ``LLMMAN_HOST`` to reach an
``llmman serve`` daemon that is not on ``127.0.0.1:17434``.

XINFERENCE_SSE_PING_ATTEMPTS_SECONDS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Server-Sent Events keepalive ping interval (seconds).
Default value is 600.

XINFERENCE_MAX_TOKENS
~~~~~~~~~~~~~~~~~~~~~
Global max tokens limit override for requests. Default is unset.

XINFERENCE_ALLOWED_IPS
~~~~~~~~~~~~~~~~~~~~~~
Restrict access to specified IPs or CIDR blocks. Default is unset (no restriction).

XINFERENCE_HTTP_REQUEST_TIMEOUT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Maximum seconds allowed to receive a complete HTTP request (headers and body).
Connections that fail to deliver a full request in time are closed, protecting
against slow-request denial-of-service attacks such as Slowloris. Streaming
responses and websockets are not affected. Set to 0 to disable.
Default value is 120.

XINFERENCE_HTTP_TIMEOUT_KEEP_ALIVE
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Idle timeout (seconds) for keep-alive connections between requests.
Default value is 5.

XINFERENCE_HTTP_LIMIT_CONCURRENCY
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Maximum number of concurrent connections or tasks the HTTP server allows
before returning 503 responses. Default is unset (unlimited).

XINFERENCE_BATCH_SIZE
~~~~~~~~~~~~~~~~~~~~~
Default batch size used by the server when batching is enabled.
Default value is 32.

For text-only ``sentence_transformers`` models, the engine may group similar
character lengths to reduce padding while preserving response order. The
internal ``encode()`` ``batch_size`` limits these groups; ``XINFERENCE_BATCH_SIZE``
controls input merging by the server.

Text-only CrossEncoder rerankers may also group pairs by combined query and
document character length to reduce padding. The native ``predict()``
``batch_size`` still limits each forward pass; short pairs stay together.

XINFERENCE_BATCH_INTERVAL
~~~~~~~~~~~~~~~~~~~~~~~~~
Default batching interval (seconds).
Default value is 0.003.

XINFERENCE_EMBEDDING_GC_INTERVAL
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Minimum interval (seconds) between full Python garbage collections during
embedding cache cleanup. Default is 60. The first cleanup collects immediately.
Set to 0 to collect on every cleanup. GPU cache release frequency is unchanged.

The cooldown also applies to token-triggered cleanup. Tensors kept alive by
Python reference cycles may remain allocated until automatic GC or a later full
collection; ``empty_cache()`` cannot release them. Set this interval to 0 for
models that require full collection on every cleanup.

XINFERENCE_RERANK_EMPTY_CACHE_COUNT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of completed inference groups between cache releases in the Sentence
Transformers and synchronous vLLM rerank engines. Default is 10. A merged group
counts once, regardless of how many requests it contains.

XINFERENCE_RERANK_GC_INTERVAL
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Minimum interval (seconds) between full Python garbage collections during
rerank cache cleanup. Default is 60. The first cache cleanup collects immediately.
Set to 0 to collect on every cache cleanup. GPU cache release still follows
``XINFERENCE_RERANK_EMPTY_CACHE_COUNT``.

Python reference cycles may keep tensors allocated until automatic GC or a
later full collection; ``empty_cache()`` cannot release them. Set this interval
to 0 for models that need full collection on every cache cleanup.

XINFERENCE_ALLOW_MULTI_REPLICA_PER_GPU
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Whether to allow multiple replicas on a single GPU.
Default value is 1 (enabled).

XINFERENCE_LAUNCH_STRATEGY
~~~~~~~~~~~~~~~~~~~~~~~~~~
GPU allocation strategy for replicas. Default is ``IDLE_FIRST_LAUNCH_STRATEGY``.

XINFERENCE_MAX_CONCURRENT_LAUNCHES
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Maximum number of model launches that can proceed concurrently on a single
worker node. When more replicas are launched than this limit, excess launches
queue and proceed as slots free up. This prevents resource exhaustion (fork
storms, disk IO saturation, GPU memory contention) that can cause heartbeat
timeouts. Default value is 5.

XINFERENCE_ENABLE_VIRTUAL_ENV
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Enable model virtual environments globally.
Default value is 1 (enabled, starting from v2.0).

XINFERENCE_VIRTUAL_ENV_SKIP_INSTALLED
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Skip packages already present in system site-packages when creating virtual environments.
Default value is 1.

XINFERENCE_CSG_TOKEN
~~~~~~~~~~~~~~~~~~~~
Authentication token for CSGHub model source.
Default is unset.

XINFERENCE_CSG_ENDPOINT
~~~~~~~~~~~~~~~~~~~~~~~
CSGHub endpoint for model source.
Default value is ``https://hub-stg.opencsg.com/``.

XINFERENCE_QWEN3_RERANK_TEMPLATE
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Enable template for Qwen3 rerank model family (0.6B, 4B, 8B,etc) globally.
Default value is 1.

XINFERENCE_LAUNCH_HISTORY_DB_PATH
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Path to the SQLite database that stores the model launch configuration history
shown in the "Launch Model" drawer of the Web UI. This store is shared across
all clients so the history is available from any browser or machine, and it is
independent of the authentication database. When authentication is enabled, each
record keeps the creator's username (``created_by``).
Default value is ``<XINFERENCE_HOME>/launch_history.db``.

XINFERENCE_MODEL_ACTOR_AUTO_RECOVER_LIMIT

Path to a static export of the web UI to serve instead of the one bundled
with the package. Default is unset (serve the bundled Next.js export).

.. versionchanged:: 3.0
   Replaces the removed ``XINFERENCE_FRONTEND_ENDPOINT`` variable; the web UI
   is now served by the Xinference server itself.

XINFERENCE_AUTH_ADVANCED
~~~~~~~~~~~~~~~~~~~~~~~~
Enable the database-backed authentication system. Default value is 1
(enabled, starting from v3.0). Set to ``0`` / ``false`` / ``no`` to run
Xinference without any authentication.
See :ref:`user_guide_auth_system`.

XINFERENCE_AUTH_DB_PATH
~~~~~~~~~~~~~~~~~~~~~~~
Path to the SQLite database that stores users, permissions, API keys, and
refresh tokens. Default value is ``<XINFERENCE_HOME>/auth/auth.db``.

XINFERENCE_AUTH_JWT_SECRET_KEY

Key used to encrypt stored API keys at rest. If unset, a key is
auto-generated and persisted at ``<XINFERENCE_HOME>/auth/encryption_key`` on
first run.

XINFERENCE_ACCESS_TOKEN_EXPIRE_MINUTES

Enable OIDC single sign-on (e.g. Keycloak). Default value is 0 (disabled).
When enabled, ``XINFERENCE_OIDC_ISSUER``, ``XINFERENCE_OIDC_CLIENT_ID``,
``XINFERENCE_OIDC_CLIENT_SECRET``, and ``XINFERENCE_OIDC_REDIRECT_URI`` are
required. See :ref:`user_guide_oidc`.

XINFERENCE_AUDIT_LOG_RETENTION_DAYS

Initial delay in seconds between Router Agent bootstrap retries. The delay uses
exponential backoff with jitter. Default value is 1.

XINFERENCE_TOKEN_ROUTER_AGENT_STARTUP_RETRY_MAX_SECONDS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Maximum delay in seconds between Router Agent bootstrap retries. Default value
is 15 and must not be lower than the initial delay.

XINFERENCE_TOKEN_ROUTER_AGENT_STARTUP_RETRY_TIMEOUT_SECONDS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Total deadline in seconds for Router Agent startup registration and initial
synchronization retries. Default value is 120.

.. _environments_media:

XINFERENCE_MEDIA_BLOCK_PRIVATE_ADDRESS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Refuse client-supplied ``image_url`` / ``video_url`` / ``audio_url`` values that
resolve to loopback, private, or link-local addresses, which otherwise let a
chat request probe the server's own network (SSRF).
Default value is ``true``.

.. versionadded:: 3.5
   Enabled by default. Requests referencing intranet hosts such as
   ``http://minio.internal/x.png`` or a Kubernetes Service DNS name now fail.
   Set this to ``false`` if the deployment is not publicly exposed and the
   server is trusted to reach those hosts.

XINFERENCE_MEDIA_ALLOW_LOCAL_PATH
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Allow ``file://`` URLs and bare filesystem paths in chat media parts. When
disabled, such values are rejected instead of being read off the server's disk.
Default value is ``false``.

.. versionadded:: 3.5
   Disabled by default. Server-local paths that previously worked
   (an ``image_url`` of ``/data/img.png``) now error. Send the media as a
   ``data:`` URI or an HTTP(S) URL, or set this to ``true`` when every API
   caller is already trusted with the server's filesystem.

XINFERENCE_MEDIA_FETCH_TIMEOUT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Total wall-clock seconds allowed for fetching one remote media URL, redirects
included. A single request's media as a whole is capped at three times this
value. Default value is 20.

XINFERENCE_MEDIA_MAX_BYTES
~~~~~~~~~~~~~~~~~~~~~~~~~~
Maximum size in bytes of one fetched media file.
Default value is 67108864 (64 MiB).

---

### Doc Source Getting Started Index (`doc/source/getting_started/index.rst`)

.. _getting_started_index:

===============
Getting Started
===============


.. toctree::
   :maxdepth: 2

   installation
   using_xinference
   logging
   using_docker_image
   using_docker_compose
   using_kubernetes
   troubleshooting
   environments
   migration_3_0
   release_notes

---

### Doc Source Getting Started Installation (`doc/source/getting_started/installation.rst`)

.. _installation:

============
Installation
============
Xinference can be installed with the :ref:`one-command installer <one_line_install>` or :ref:`pip `. Choose the installer for a managed tool environment, or pip to use an existing Python environment.

.. _one_line_install:

One-command installation
~~~~~~~~~~~~~~~~~~~~~~~~

.. _one-command-installation-and-startup:

Installation and startup
------------------------
The installer prepares uv, Python 3.12, and an isolated Xinference tool environment, then starts the local server in the foreground. Press Ctrl+C to stop it. On Linux and macOS::

   curl -fsSL https://raw.githubusercontent.com/xorbitsai/inference/main/scripts/install.sh | sh

On Windows, run this in PowerShell::

   irm https://raw.githubusercontent.com/xorbitsai/inference/main/scripts/install.ps1 | iex

Set ``XINFERENCE_START=0`` to install without starting. Use ``XINFERENCE_VERSION`` to pin a release, ``XINFERENCE_PYTHON`` to select Python, ``XINFERENCE_EXTRAS`` to preinstall backends, and ``XINFERENCE_TOOL_DIR`` to change the uv tool store. ``XINFERENCE_HOME`` controls persistent model data; it is separate from the tool environment.

On Linux and Windows, ``XINFERENCE_BACKEND=auto`` selects PyTorch for the detected GPU driver or CPU. Override it with ``cpu``, ``cu128``, or another backend supported by uv. macOS uses its native PyTorch wheel. Recent PyTorch wheels require Apple Silicon; Intel Macs may resolve older dependencies or fail installation. Model virtual environments install engine dependencies on demand. Engine platform restrictions still apply, including Linux-only vLLM and Apple-Silicon-only MLX.

To run without a persistent tool installation, use ``uvx``. This requires uv; the first invocation downloads the Python environment and dependencies::

   uvx --python 3.12 --from xinference xinference-local

Use a recent uv supporting ``uv tool install --torch-backend`` (tested with uv 0.11.26). Services require a persistent installation; uvx environments can be removed by cache cleanup.

Upgrading an installation
-------------------------
.. note::

   Starting with v4.2.0, administrators can upgrade a running deployment from the Web UI: click the version at the bottom left, check for updates, then choose **Install and upgrade**. Existing models continue serving while components are replaced. See :ref:`zero_downtime_upgrade` for requirements and supported deployments.

Run the same one-command installer again to update to the latest stable release. Set ``XINFERENCE_VERSION`` each time to select a specific release, including a downgrade. Without this variable, a previously pinned installation also updates to the latest stable release. If the selected version, Python, extras, and PyTorch backend are unchanged, the installer keeps the existing environment and does not restart a running service.

The default ``XINFERENCE_SERVICE=auto`` detects an existing managed service for this tool environment. Upgrades preserve its service account, address, port, data directory, and registration. Python, extras, and the PyTorch backend are also retained unless explicitly overridden. Run as the original installation account and reuse any custom ``XINFERENCE_TOOL_DIR`` and ``UV_TOOL_BIN_DIR``. Stop a foreground server before updating it.

During an upgrade, the installer downloads and checks the candidate environment before stopping a service. It then backs up the complete previous environment, switches versions, and waits for the service to become ready. If installation or service startup fails, it restores the previous environment and the service's original running or stopped state. Preparing the candidate and keeping the full backup require additional disk space. On a first installation, failed startup removes a newly registered service and keeps the installed package for retrying. Model data and caches are preserved; models that were running must be launched again after a service restart. Foreground startup does not perform a service readiness check or automatic startup rollback.

Concurrent installers for the same tool store are rejected. If an upgrade is interrupted, the next run restores the saved environment before trying again. Keep the recovery files in the tool store until recovery completes. ``XINFERENCE_TIMEOUT`` sets the service readiness timeout in seconds (default 120). With ``XINFERENCE_START=0``, an updated service remains stopped.

The installer checks the candidate's service installer API before stopping a service. An incompatible release cannot be managed in service mode. Same-version checks still prepare a candidate to resolve the requested Python, extras, and backend. Installed dependencies are constrained to that candidate, so use this installer for upgrades; ``uv tool upgrade xinference`` retains these constraints.

Do not run ``xinference service`` commands while the installer is running; they do not share its lock. If a service was uninstalled after an interrupted upgrade, recovery restores the package without recreating the service. For manual recovery, stop processes using the tool, read ``.xinference-transaction.json`` in the tool store, copy its backup's ``environment`` and ``shims`` back to their original tool and command directories, and remove the journal only after restoration succeeds.

System services
---------------
After installing with pip, Conda, or uv, register and start a local service with::

   xinference service install --start
   xinference service status
   xinference service logs
   xinference service restart
   xinference service stop
   xinference service uninstall

Linux uses a systemd user service. To start it at boot without logging in, enable lingering for the account with ``loginctl enable-linger USER``. macOS uses a LaunchAgent that starts at login. For a systemd system service or macOS LaunchDaemon, use an elevated terminal, an absolute command path, and ``--system`` before the action::

   sudo /absolute/path/to/xinference service --system install --user USER --start
   sudo /absolute/path/to/xinference service --system status
   sudo /absolute/path/to/xinference service --system uninstall

For ``--user USER``, that account must be able to traverse the Python installation's parent directories and execute its interpreter. The environment owner can modify code executed as the service account. Use a shared installation with appropriate ownership and permissions when these accounts differ.

Windows services currently require x86-64 Windows and an Administrator PowerShell terminal and run as LocalSystem. The installer downloads a pinned, checksum-verified WinSW wrapper. Service control files and the working directory use ``%ProgramFiles%\Xinference\service``. Use an administrator-controlled Python installation and model data because the service executes model code with its account's privileges. Service data defaults to ``%PROGRAMDATA%\Xinference\data``. Linux and macOS default to the runtime account's ``~/.xinference``. Override the data directory with ``--home`` and the address with ``--host`` and ``--port`` on ``service install``.

The one-command Windows system installer uses the installing user's uv tool store and does not restrict runtime or data ACLs. This default is not an administrator-only Python installation. Protect both Python (including packages) and model data before starting a LocalSystem service, or use foreground mode. Any account able to modify them can execute code with the service account's privileges.

A macOS user service requires a logged-in graphical session; use system mode on headless hosts. The launchd ``console.log`` does not rotate automatically.

For one-command service installation, set ``XINFERENCE_SERVICE=user`` on Linux/macOS or ``XINFERENCE_SERVICE=system`` on any supported platform. System mode uses sudo on Linux/macOS after installing as your account, and requires an Administrator Windows terminal. ``XINFERENCE_START=0`` registers without starting. ``XINFERENCE_HOST`` and ``XINFERENCE_PORT`` default to ``127.0.0.1`` and ``9997``. Service mode requires a Xinference release containing the service CLI::

   curl -fsSL https://raw.githubusercontent.com/xorbitsai/inference/main/scripts/install.sh | XINFERENCE_SERVICE=user sh

   curl -fsSL https://raw.githubusercontent.com/xorbitsai/inference/main/scripts/install.sh | XINFERENCE_SERVICE=system sh

   $env:XINFERENCE_SERVICE='system'; irm https://raw.githubusercontent.com/xorbitsai/inference/main/scripts/install.ps1 | iex

Services restart after failures. Starting waits for the cluster's ``/status`` response; a failed readiness check stops a newly started service. Repeated installation reuses the same configuration. Stop and uninstall before changing the installation environment or service settings. Uninstall preserves model data and logs. This service interface manages single-machine local mode.

.. _pip_installation:

Installation with pip
~~~~~~~~~~~~~~~~~~~~~
Install Xinference with ``pip`` on Linux, Windows, or macOS::

   pip install xinference

Automatic engine installation
-----------------------------
Model virtual environments are enabled by default. When you launch a model,
Xinference prepares a virtual environment for that model and engine and
automatically installs the dependencies declared by its configuration.
You do not need to install ``xinference[all]`` or an engine extra first.

This applies to engines such as Transformers, vLLM, SGLang, llama.cpp, and MLX.
Choose a supported engine when launching the model; Xinference manages its
Python dependencies. The engine's operating system, hardware, and driver
requirements still apply.

The first launch may take longer while packages are downloaded and installed.
Later launches reuse the prepared environment when its dependencies are unchanged.
The Worker needs access to the configured package indexes, or a prepared offline
package source. See :ref:`model_virtual_env` for dependency overrides and local
wheel sources, and :ref:`using_docker_compose` for offline deployments.

The following sections describe the available engines. Manual installation is
covered in :ref:`manual_backend_installation` for deployments that disable model
virtual environments.


.. _inference_backend:

Transformers Backend
--------------------
PyTorch (transformers) supports the inference of most state-of-art models. It is the default backend for models in PyTorch format.

Notes:

- The transformers engine supports ``pytorch`` / ``gptq`` / ``awq`` / ``bnb`` / ``fp4`` formats.
- FP4 format requires ``transformers`` with ``FPQuantConfig`` support. Use
  :ref:`model_virtual_env` to override dependencies for a model when needed.


vLLM Backend
------------
vLLM is a fast and easy-to-use library for LLM inference and serving. Xinference will choose vLLM as the backend to achieve better throughput when the following conditions are met:

- The model format is ``pytorch``, ``gptq``, ``awq``, ``fp4``, ``fp8`` or ``bnb``.
- When the model format is ``pytorch``, the quantization is ``none``.
- When the model format is ``awq``, the quantization is ``Int4``.
- When the model format is ``gptq``, the quantization is ``Int3``, ``Int4`` or ``Int8``.
- The system is Linux and has at least one CUDA device
- The model family (for custom models) / model name (for builtin models) is within the list of models supported by vLLM

Currently, supported models include:

.. vllm_start

- ``code-llama``, ``code-llama-instruct``, ``code-llama-python``, ``deepseek``, ``deepseek-chat``, ``deepseek-coder``, ``deepseek-coder-instruct``, ``deepseek-r1-distill-llama``, ``HuatuoGPT-o1-LLaMA-3.1``, ``llama-2``, ``llama-2-chat``, ``llama-3``, ``llama-3-instruct``, ``llama-3.1``, ``llama-3.1-instruct``, ``llama-3.3-instruct``, ``minicpm5-1b``, ``tiny-llama``, ``Yi``, ``Yi-1.5``, ``Yi-1.5-chat``, ``Yi-1.5-chat-16k``, ``Yi-200k``, ``Yi-chat``
- ``codestral-v0.1``, ``mistral-instruct-v0.1``, ``mistral-instruct-v0.2``, ``mistral-instruct-v0.3``, ``mistral-large-instruct``, ``mistral-nemo-instruct``, ``mistral-v0.1``, ``openhermes-2.5``, ``seallm_v2``
- ``Baichuan-M2``, ``codeqwen1.5``, ``codeqwen1.5-chat``, ``deepseek-r1-distill-qwen``, ``DianJin-R1``, ``fin-r1``, ``HuatuoGPT-o1-Qwen2.5``, ``KAT-V1``, ``marco-o1``, ``qwen1.5-chat``, ``qwen2-instruct``, ``qwen2.5``, ``qwen2.5-coder``, ``qwen2.5-coder-instruct``, ``qwen2.5-instruct``, ``qwen2.5-instruct-1m``, ``qwenLong-l1``, ``QwQ-32B``, ``QwQ-32B-Preview``, ``seallms-v3``, ``skywork-or1``, ``skywork-or1-preview``, ``vibethinker``, ``XiYanSQL-QwenCoder-2504``
- ``llama-3.2-vision``, ``llama-3.2-vision-instruct``
- ``baichuan-2``, ``baichuan-2-chat``
- ``InternLM2ForCausalLM``
- ``qwen-chat``
- ``mixtral-8x22B-instruct-v0.1``, ``mixtral-instruct-v0.1``, ``mixtral-v0.1``
- ``cogagent``
- ``glm-edge-chat``, ``glm4-chat``, ``glm4-chat-1m``
- ``codegeex4``, ``glm-4v``
- ``qwen3.8-max``
- ``seallm_v2.5``
- ``orion-chat``
- ``qwen1.5-moe-chat``, ``qwen2-moe-instruct``
- ``CohereForCausalLM``
- ``deepseek-v2-chat``, ``deepseek-v2-chat-0628``, ``deepseek-v2.5``, ``deepseek-vl2``
- ``deepseek-prover-v2``, ``deepseek-r1``, ``deepseek-r1-0528``, ``deepseek-v3``, ``deepseek-v3-0324``, ``Deepseek-V3.1``, ``moonlight-16b-a3b-instruct``
- ``deepseek-r1-0528-qwen3``, ``qwen3``
- ``minicpm3-4b``
- ``internlm3-instruct``
- ``gemma-3-1b-it``
- ``glm4-0414``
- ``minicpm-2b-dpo-bf16``, ``minicpm-2b-dpo-fp16``, ``minicpm-2b-dpo-fp32``, ``minicpm-2b-sft-bf16``, ``minicpm-2b-sft-fp32``, ``minicpm4``
- ``Ernie4.5``
- ``Qwen3-Coder``, ``Qwen3-Instruct``, ``Qwen3-Thinking``
- ``glm-4.5``, ``GLM-4.6``, ``GLM-4.7``
- ``gpt-oss``
- ``seed-oss``
- ``Qwen3-Next-Instruct``, ``Qwen3-Next-Thinking``
- ``DeepSeek-V3.2``, ``DeepSeek-V3.2-Exp``
- ``MiniMax-M2``, ``MiniMax-M2.5``, ``MiniMax-M2.7``
- ``GLM-4.7-Flash``
- ``glm-5``, ``glm-5.1``, ``glm-5.2``, ``glm-5.3``
- ``DeepSeek-V4-Flash``, ``DeepSeek-V4-Flash-0731``, ``DeepSeek-V4-Pro``
- ``Hy-MT2-1.8B``, ``Hy-MT2-7B``
- ``Hy-MT2-30B-A3B``

.. vllm_end

With model virtual environments enabled, Xinference installs the configured
vLLM dependencies at launch. No separate vLLM or FlashInfer installation command
is required for this path.

.. _installation_gguf:

Llama.cpp Backend
-----------------
Xinference supports models in ``gguf`` format via ``xllamacpp``.
`xllamacpp <https://github.com/xorbitsai/xllamacpp>`_ is developed by Xinference team,
and is the sole backend for llama.cpp since v1.6.0.

.. warning::

    Since Xinference v1.5.0, ``llama-cpp-python`` is deprecated.
    Since Xinference v1.6.0, ``llama-cpp-python`` has been removed.

With per-model virtual environments enabled, Xinference 3.0 automatically
selects the matching ``xllamacpp`` GPU wheel for supported CUDA versions; see
:ref:`user_guide_backends`. For a manual installation into the process
environment, refer to https://github.com/xorbitsai/xllamacpp.

SGLang Backend
--------------
SGLang has a high-performance inference runtime with RadixAttention. It significantly accelerates the execution of complex LLM programs by automatic KV cache reuse across multiple calls. And it also supports other common techniques like continuous batching and tensor parallelism.

Select ``SGLang`` as the engine when launching a supported model. Xinference
installs the configured SGLang dependencies in that model's virtual environment;
you do not need to install ``xinference[sglang]`` separately.


MLX Backend
-----------
MLX-lm is designed for Apple silicon users to run LLM efficiently.

Xinference installs the configured MLX dependencies automatically when you launch
a supported model with the MLX engine.

Optional components
-------------------

Token Router
^^^^^^^^^^^^
Token Router is an optional component for routing requests across model replicas.
It is not required to install Xinference or launch models. If your deployment
uses the Token Router Runtime or Router Agent, install the ``router`` extra in
the environment where those processes run::

   pip install "xinference[router]"

When installing from a source checkout, use::

   uv pip install -e ".[router]"

Installing the package creates the ``xinference-router`` and
``xinference-router-agent`` commands. Installing the extra provides their
dependencies; start the required processes separately.

.. _manual_backend_installation:

Manual backend installation
---------------------------
If you disable model virtual environments with
``XINFERENCE_ENABLE_VIRTUAL_ENV=0`` or ``--disable-virtual-env`` at model launch,
install the required backend dependencies in the Xinference Worker environment
yourself. Engine extras such as ``transformers``, ``vllm``, ``sglang``,
``llama_cpp``, and ``mlx`` remain available for this use case, for example:

.. code-block:: bash

   pip install "xinference[transformers]"

Some hardware or quantization configurations require additional installation
steps. Follow the selected backend's installation instructions for compatible
packages and GPU support. Avoid installing unrelated engines together, since
their dependencies may conflict.

.. only:: zh_cn

   Other Platforms
   ---------------

   * :ref:`Ascend NPU <installation_npu>`

---

### Doc Source Getting Started Installation Npu (`doc/source/getting_started/installation_npu.rst`)

.. _installation_npu:


=================================
Installation Guide for Ascend NPU
=================================
Xinference can run on Ascend NPU, follow below instructions to install.

.. warning::

    The open-source version relies on Transformers for inference,
    which can be slow on chips like 310p3. We provide an enterprise version that supports the MindIE engine,
    offering better performance and compatibility for Ascend NPU.
    Refer to `Xinference Enterprise <https://xinference.io>`_


Installing PyTorch and Ascend extension for PyTorch
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Install PyTorch CPU version and corresponding Ascend extension.

Take PyTorch v2.1.0 as example.

  .. code-block:: bash

    pip3 install torch==2.1.0 torchvision==0.16.0 --index-url https://download.pytorch.org/whl/cpu

Then install `Ascend extension for PyTorch <https://github.com/Ascend/pytorch>`_.

  .. code-block:: bash

    pip3 install 'numpy<2.0'
    pip3 install decorator
    pip3 install torch-npu==2.1.0.post3

Running below command to see if it correctly prints the Ascend NPU count.

.. code-block:: bash

    python -c "import torch; import torch_npu; print(torch.npu.device_count())"

Installing Xinference
~~~~~~~~~~~~~~~~~~~~~

.. code-block:: bash

    pip3 install xinference

Now you can use xinference according to :ref:`doc <using_xinference>`.
``Transformers`` backend is the only available engine supported for Ascend NPU for open source version.

Enterprise Support
~~~~~~~~~~~~~~~~~~
If you encounter any performance or other issues for Ascend NPU, please reach out to us
via `link <https://xinference.io>`_.

---

### Doc Source Getting Started Logging (`doc/source/getting_started/logging.rst`)

.. _logging:

=====================
Logging in Xinference
=====================

Configure Log Level
###################
You can configure the log level with the ``--log-level`` option.
For example, starting a local cluster with ``DEBUG`` log level:

.. code-block:: bash

  xinference-local --log-level debug


Log Files
#########
Xinference supports log rotation of log files.
By default, logs rotate when they reach 100MB (maxBytes), and up to 30 backup files (backupCount) are kept.
Note that the log level configured above takes effect in both the command line logs and the log files.

Environment Variables
#####################
Xinference provides several environment variables to control logging behavior:

- ``XINFERENCE_LOG_CONSOLE``: Enable or disable console output (default: ``true``).
  When set to ``false``, logs are written only to files, and tqdm progress bars are captured and sampled.
- ``XINFERENCE_LOG_FORMAT``: Log format, either ``text`` (default) or ``json``.
- ``XINFERENCE_LOG_DOWNLOAD_PROGRESS``: Control how download progress bars are logged when ``XINFERENCE_LOG_CONSOLE=false``.
  Valid values are ``sampled`` (default, logs at 25/50/75/100% per file), ``full`` (logs every frame), or ``off`` (no progress logs).

Example usage:

.. code-block:: bash

  # Disable console output, log download progress at sampling points
  XINFERENCE_LOG_CONSOLE=false XINFERENCE_LOG_DOWNLOAD_PROGRESS=sampled xinference-local

  # Disable console output, log every download progress frame
  XINFERENCE_LOG_CONSOLE=false XINFERENCE_LOG_DOWNLOAD_PROGRESS=full xinference-local

  # Disable console output, no download progress logs
  XINFERENCE_LOG_CONSOLE=false XINFERENCE_LOG_DOWNLOAD_PROGRESS=off xinference-local

Log Directory Structure
***********************
Application logs are written to ``<XINFERENCE_LOG_DIR>/xinference.log``. By
default, ``XINFERENCE_LOG_DIR`` is ``<XINFERENCE_HOME>/logs``. If
``XINFERENCE_POD_NAME`` is set, the file is named
``xinference-<XINFERENCE_POD_NAME>.log`` instead. Local deployment combines
Supervisor and Worker logs in this file. In distributed deployment, each node
writes to its own local log directory. Rotated files stay beside the active file.

Web UI runtime logs
###################

Open **Monitoring Management > Log Center > Runtime logs** to view recent
application logs and errors without Elasticsearch. The page reads the active
log file on the selected node and refreshes every two seconds. It works when
Xinference runs directly with Python or inside Docker. If Elasticsearch is
configured, the Log Center also offers an Indexed logs tab for historical
search.

The runtime view shows the latest part of the active file and keeps a bounded
amount of text in the browser. Search filters only loaded text. Output written
directly to stdout or stderr outside Xinference logging, and failures before
the API starts, may require the terminal or ``docker logs``. Log access uses
the ``logs:list`` permission when authentication is enabled.


Token Router logging
####################

The independent ``xinference-router`` service uses the same Xinference file
formatters and rotation handlers as the Supervisor and Worker processes. A
production systemd deployment can use the following non-sensitive settings in
``/etc/xinference/router.env``:

.. code-block:: ini

  XINFERENCE_TOKEN_ROUTER_LOG_LEVEL=INFO
  XINFERENCE_TOKEN_ROUTER_ACCESS_LOG=false
  XINFERENCE_LOG_FORMAT=json
  XINFERENCE_LOG_CONSOLE=false
  XINFERENCE_LOG_DIR=/data/inference/logs/router
  XINFERENCE_LOG_ROTATION=daily+size
  XINFERENCE_LOG_RETENTION_DAYS=30
  XINFERENCE_LOG_MAX_BYTES=104857600
  XINFERENCE_LOG_BACKUP_COUNT=300

With these settings, Router application logs are written to
``/data/inference/logs/router/xinference.log``. The directory must exist and be
writable by the Router service account. Rotation is managed by Xinference; do
not apply an additional ``logrotate``/``copytruncate`` rule to the same file.

The Router emits structured lifecycle, configuration, routing decision,
completion, rejection, and backend-error events. Routing events include both
the requested virtual model and the selected physical backend model UID when
available. Request bodies, prompt/message content, response bodies,
Authorization headers, API keys, and control-plane tokens are not logged.

Uvicorn access logs are disabled by default because they duplicate high-volume
request information. Set ``XINFERENCE_TOKEN_ROUTER_ACCESS_LOG=true`` only when
access-log diagnostics are required. Uvicorn error logs remain enabled and use
the same Xinference logging configuration. In systemd deployments, journal
output should remain enabled as a fallback for process lifecycle messages and
failures that occur before application logging is initialized.

Model request body logging
##########################

Xinference can write inference request metadata and request bodies to a separate
JSON-lines file. This diagnostic feature is disabled by default because prompts,
media references, and other request values may contain sensitive data. Enable it
only on trusted systems with appropriate access controls and retention policies.
Authentication and authorization failures do not persist request bodies.
Multipart uploads record field values and original filenames, but not uploaded
file bytes. Requests larger than the configured capture limit, or requests
without a known size, record an omission reason instead of the body.
Each request event also records its producing process identity in ``role``,
``address``, ``node``, ``module``, and ``pid``, together with an
``api_protocol`` value of ``openai``, ``anthropic``, or ``xinference``.

The following environment variables configure the feature:

- ``XINFERENCE_MODEL_REQUEST_LOG_ENABLED``: Enable request body logging (default: ``false``).
- ``XINFERENCE_MODEL_REQUEST_LOG_FILE``: Log filename or absolute path (default: ``model_request.log``).
- ``XINFERENCE_MODEL_REQUEST_LOG_BODY_MAX_BYTES``: Maximum captured request size (default: ``16777216``). Set to ``-1`` to disable the size limit; this is not recommended for internet-facing deployments.
- ``XINFERENCE_MODEL_REQUEST_LOG_RETENTION_DAYS``: Maximum age of rotated files in days (default: ``7``).
- ``XINFERENCE_MODEL_REQUEST_LOG_MAX_BYTES``: Size-based rotation threshold (default: ``1073741824``).
- ``XINFERENCE_MODEL_REQUEST_LOG_BACKUP_COUNT``: Maximum number of rotated files (default: ``7``).

Each inference response includes ``X-Request-ID``. Xinference preserves a valid
caller-provided ``request-id`` or ``x-request-id`` value (in that order), or
generates a UUID. The same correlation ID is attached to
Supervisor model lookup logs without replacing model operation request IDs used
for cancellation or progress tracking.

For streaming responses, the terminal request log separates HTTP delivery from
stream execution with ``http_success``, ``stream_completed``,
``stream_outcome``, and, for non-successful outcomes, ``failure_origin``. A
response can therefore have HTTP status 200 while ``success`` and
``stream_completed`` are false. Endpoints report swallowed generator failures
through request state; the logging layer does not parse, buffer, or log stream
chunks. Failure origins are ``model_generator``, ``upstream``, ``protocol``,
``client``, or ``server``.

Correlation metadata is propagated independently across REST API, Supervisor,
Worker, ModelActor, PD, cancellation, and progress actor calls. Actor log records
can include ``correlation_id``, ``operation_request_id``, ``actor_call_id``, and
``parent_call_id``. The correlation ID is observability-only; the operation
request ID retains its existing cancellation, progress, batching, and backend
abort semantics. Xinference consumes the internal metadata envelope at actor
boundaries and does not pass it to model implementations.

---

### Doc Source Getting Started Migration 3 0 (`doc/source/getting_started/migration_3_0.rst`)

.. _migration_3_0:

============================
Migrating to Xinference 3.0+
============================

Xinference 3.0 modernizes authentication, the web UI, and container
deployment. Most workloads keep working unchanged, but a few behaviors and
flags were removed or renamed. This page lists every breaking change and how
to adapt.

Authentication is enabled by default
=====================================

Starting with v3.0, the database-backed authentication system is **enabled by
default**. A fresh deployment requires a login or an API key for every API
call.

* On first startup, copy the setup token from the server startup log. Enter it
  in the web UI, or include it as ``setup_token`` alongside ``username`` and
  ``password`` when calling ``POST /v1/admin/setup``. See
  :ref:`user_guide_auth_system` for the full flow.
* To restore the previous unauthenticated behavior, set:

  .. code-block:: bash

     export XINFERENCE_AUTH_ADVANCED=false

* Existing clients and scripts that talk to an authenticated server must now
  log in (``xinference login`` / ``client.login()``) or pass an API key
  (``--api-key`` / ``api_key=`` / ``Authorization: Bearer <key>``).

The legacy ``--auth-config`` system is removed
===============================================

The in-memory JSON authentication system configured through the
``--auth-config`` flag of ``xinference-local`` and ``xinference-supervisor``
has been removed, together with its static ``auth_config.json`` format. The
flag no longer exists.

To carry your old users and API keys over to the new database-backed system,
use the migration command:

.. code-block:: bash

   xinference-migrate-auth \
     --auth-config /path/to/old/auth_config.json \
     --db-path ~/.xinference/auth/auth.db \
     --encryption-key <your-encryption-key> \
     --dry-run   # preview first, then run again without --dry-run

The value passed to ``--encryption-key`` must be the **same key** that the
upgraded server uses at runtime: either the value of
``XINFERENCE_AUTH_ENCRYPTION_KEY`` or the value persisted in
``<XINFERENCE_HOME>/auth/encryption_key``. It is not a one-time migration
secret. If a different key is used, the migrated API-key hashes can still
authenticate requests, but Xinference cannot decrypt or reveal the stored
plaintext keys.

If an admin password is ever lost, an operator with shell access can reset it
offline with ``xinference-reset-auth-password`` — see
:ref:`user_guide_auth_system`.

Permission scope renames
=========================

Several fine-grained permission scopes were consolidated and renamed:

.. list-table::
   :header-rows: 1

   * - Legacy scope
     - Replacement
   * - ``models:start``
     - ``models:write``
   * - ``models:stop``
     - ``models:write``
   * - ``models:add``
     - ``models:register``
   * - ``admin`` (for log routes)
     - ``logs:list``
   * - ``admin`` (required by the legacy API-key reveal route)
     - ``keys:manage`` (list any user's keys; update, delete, reveal, and
       manage per-key model permissions)

API key creation remains separate: ``keys:create`` is required to create a
key, and ``keys:manage`` is additionally required when creating one for
another user.

Tokens and API keys that carry the legacy scope names keep working: the
server transparently maps legacy names to their replacements. This
compatibility mapping is deprecated and will be removed in a future release,
so update any automation that grants permissions to use the new names.

New default web UI (Next.js)
=============================

The web UI has been rewritten in Next.js and is now the default; the legacy
React UI has been removed. The UI is exported statically at build time and
served by the Xinference server itself — no separate frontend process is
needed.

* The ``XINFERENCE_FRONTEND_ENDPOINT`` environment variable has been
  **removed**. To serve a custom frontend build, point
  ``XINFERENCE_FRONTEND_DIST_DIR`` at its directory instead.
* The old ``/ui/`` path redirects to ``/``. Opening ``/`` takes you to the
  model launch page.

Built-in Gradio demo pages are removed
=======================================

The per-model Gradio demo UI (previously mounted at ``/{model_uid}``) and the
``/v1/ui/*`` endpoints behind it have been removed, along with the ``gradio``
dependency. To interact with a running model, use the web UI, the
OpenAI-compatible API, or the Python client instead.

The official GPU image requires CUDA 13 and installs engines on demand
=======================================================================

The official 3.0 GPU image is now a slim image based on
``nvidia/cuda:13.0.2-devel-ubuntu22.04``. It requires an NVIDIA driver version
of **580 or later**. The image still includes Python 3.12, the shared CUDA
PyTorch stack, and the Transformers engine, but it no longer pre-installs
vLLM, SGLang, llama.cpp, or other optional inference engines.

When a model needs one of those engines, Xinference installs it into a
per-model virtual environment on first launch. The first launch therefore
takes longer and requires access to PyPI or a compatible private package
mirror; later launches reuse the environment.

For offline or air-gapped Compose deployments, pull or transfer the matching
``xprobe/xinference-pypiserver`` image and pin it to the same release tag as
``xprobe/xinference``. Alternatively, supply your own wheel directory with
the ``docker-compose.byo-wheels.yml`` override. See
:ref:`using_docker_image` and :ref:`using_docker_compose` for the complete
requirements.

Strict Qwen3-family system-message ordering
============================================

For model families whose chat template requires the system message to come
first (including Ornith-1.0-35B, qwen3.5, qwen3.6, and Nex-N2), Xinference now
validates message order before dispatching a request. A ``system`` message at
any position other than ``messages[0]`` returns HTTP 400. Move all system
instructions into a single leading system message before upgrading.

Docker Compose requirements
============================

The bundled Compose files dropped the deprecated ``version:`` field and use
newer Compose features, so deploying with them requires **Docker Compose
v2.24.4 or later**. See :ref:`using_docker_compose`.

---

### Doc Source Getting Started Release Notes (`doc/source/getting_started/release_notes.rst`)

.. _release_ntoes:

Release Notes
=============

.. _release_4_0_overview:

Xinference 4.0 overview
-----------------------

The v4.0.0 community release expands cache reuse, heterogeneous
deployment, and runtime model management. The highlights below summarize the
major changes; detailed notes for individual versions follow this overview.

* **Multi-engine Xavier and P/D.** SGLang and MLX replicas gain prefix KV
  cache sharing. P/D deployments can combine vLLM, SGLang, and MLX, including
  NVIDIA and Apple silicon workers. Native NIXL backends are available for
  vLLM and SGLang; vLLM Xavier adds direct GPU handoff and tiered history.
  See :ref:`user_guide_vllm_enhancement` and :ref:`user_guide_pd_separation`.
* **Reconfigure without reloading weights.** Opt-in GPU weight caching lets
  vLLM and SGLang rebuild the engine with updated supported parameters while
  retaining the weights. Current requests drain before the change, and the
  model UID stays the same. See :ref:`launch_weight_cache_reload`.
* **Distributed vLLM V1 inference.** Supported models can combine tensor
  parallelism and pipeline parallelism across workers through Xinference's
  xoscar executor, without a separate Ray cluster. See
  :ref:`distributed_inference`.
* **OpenAI-compatible API additions.** A stateless Responses API supports
  clients using ``/v1/responses``. vLLM, SGLang, and MLX also expose reused
  prompt-token counts in usage responses when reported by the engine. See
  :ref:`openai_responses_client` and :ref:`user_guide_client_api`.
* **Unified logging.** The Web UI Log Center provides recent runtime logs
  and, with Elasticsearch configured, historical indexed logs. Request IDs
  connect API, audit, and runtime events. See :ref:`logging` and
  :ref:`user_guide_audit_security`.
* **Installation and local services.** The cross-platform installer and
  service commands simplify setup and local service management on Linux,
  macOS, and Windows. Engine dependencies continue to install on demand in
  per-model virtual environments. See :ref:`one_line_install` and
  :ref:`model_launching_virtualenv`.

If upgrading from a version earlier than 3.0, read :ref:`migration_3_0` for
authentication, Web UI, and container deployment changes. Check each feature
guide for its engine and hardware requirements.

Detailed version notes
----------------------

This page provides a version-by-version index of Xinference release notes.
For detailed updates, please visit the corresponding links below.

+-----------------+--------------------------------------------------------------------------------+
| Version         | Release Notes                                                                  |
+=================+================================================================================+
| v4.1.0          | `View release notes <https://xinference.co/release_notes/v4.1.0.html>`_        |
+-----------------+--------------------------------------------------------------------------------+
| v4.0.0          | `View release notes <https://xinference.co/release_notes/v4.0.0.html>`_        |
+-----------------+--------------------------------------------------------------------------------+
| v3.5.0          | `View release notes <https://xinference.co/release_notes/v3.5.0.html>`_        |
+-----------------+--------------------------------------------------------------------------------+
| v3.4.0          | `View release notes <https://xinference.co/release_notes/v3.4.0.html>`_        |
+-----------------+--------------------------------------------------------------------------------+
| v3.3.0          | `View release notes <https://xinference.co/release_notes/v3.3.0.html>`_        |
+-----------------+--------------------------------------------------------------------------------+
| v3.2.0          | `View release notes <https://xinference.co/release_notes/v3.2.0.html>`_        |
+-----------------+--------------------------------------------------------------------------------+
| v3.1.0          | `View release notes <https://xinference.co/release_notes/v3.1.0.html>`_        |
+-----------------+--------------------------------------------------------------------------------+
| v3.0.0          | `View release notes <https://xinference.co/release_notes/v3.0.0.html>`_        |
+-----------------+--------------------------------------------------------------------------------+
| v2.12.0         | `View release notes <https://xinference.co/release_notes/v2.12.0.html>`_       |
+-----------------+--------------------------------------------------------------------------------+
| v2.11.0         | `View release notes <https://xinference.co/release_notes/v2.11.0.html>`_       |
+-----------------+--------------------------------------------------------------------------------+
| v2.10.0         | `View release notes <https://xinference.co/release_notes/v2.10.0.html>`_       |
+-----------------+--------------------------------------------------------------------------------+
| v2.9.0          | `View release notes <https://xinference.co/release_notes/v2.9.0.html>`_        |
+-----------------+--------------------------------------------------------------------------------+
| v2.8.0          | `View release notes <https://xinference.co/release_notes/v2.8.0.html>`_        |
+-----------------+--------------------------------------------------------------------------------+
| v2.7.0          | `View release notes <https://xinference.co/release_notes/v2.7.0.html>`_        |
+-----------------+--------------------------------------------------------------------------------+
| v2.5.0          | `View release notes <https://xinference.co/release_notes/v2.5.0.html>`_        |
+-----------------+--------------------------------------------------------------------------------+
| v2.4.0          | `View release notes <https://xinference.co/release_notes/v2.4.0.html>`_        |
+-----------------+--------------------------------------------------------------------------------+
| v2.3.0          | `View release notes <https://xinference.co/release_notes/v2.3.0.html>`_        |
+-----------------+--------------------------------------------------------------------------------+
| v2.2.0          | `View release notes <https://xinference.co/release_notes/v2.2.0.html>`_        |
+-----------------+--------------------------------------------------------------------------------+
| v2.1.0          | `View release notes <https://xinference.co/release_notes/v2.1.0.html>`_        |
+-----------------+--------------------------------------------------------------------------------+
| v2.0.0          | `View release notes <https://xinference.co/release_notes/v2.0.0.html>`_        |
+-----------------+--------------------------------------------------------------------------------+
| v1.17.0         | `View release notes <https://xinference.co/release_notes/v1.17.0.html>`_       |
+-----------------+--------------------------------------------------------------------------------+
| v1.16.0         | `View release notes <https://xinference.co/release_notes/v1.16.0.html>`_       |
+-----------------+--------------------------------------------------------------------------------+
| v1.15.0         | `View release notes <https://xinference.co/release_notes/v1.15.0.html>`_       |
+-----------------+--------------------------------------------------------------------------------+
| v1.14.0         | `View release notes <https://xinference.co/release_notes/v1.14.0.html>`_       |
+-----------------+--------------------------------------------------------------------------------+
| v1.13.0         | `View release notes <https://xinference.co/release_notes/v1.13.0.html>`_       |
+-----------------+--------------------------------------------------------------------------------+
| v1.12.0         | `View release notes <https://xinference.co/release_notes/v1.12.0.html>`_       |
+-----------------+--------------------------------------------------------------------------------+
| v1.11.0.post1   | `View release notes <https://xinference.co/release_notes/v1.11.0.post1.html>`_ |
+-----------------+--------------------------------------------------------------------------------+
| v1.10.1         | `View release notes <https://xinference.co/release_notes/v1.10.1.html>`_       |
+-----------------+--------------------------------------------------------------------------------+

----

For older versions and source history, see our GitHub releases page:
https://github.com/xorbitsai/inference/releases

---

### Doc Source Getting Started Troubleshooting (`doc/source/getting_started/troubleshooting.rst`)

.. _troubleshooting:

===============
Troubleshooting
===============


No huggingface repo access
==========================

Sometimes, you may face errors accessing huggingface models, such as the following message when accessing `llama2`:

.. code-block:: text

   Cannot access gated repo for url https://huggingface.co/api/models/meta-llama/Llama-2-7b-hf.
   Repo model meta-llama/Llama-2-7b-hf is gated. You must be authenticated to access it.

This typically indicates either a lack of access rights to the repository or missing huggingface access tokens. 
The following sections provide guidance on addressing these issues.

Get access to the huggingface repo
----------------------------------

To obtain access, navigate to the desired huggingface repository and agree to its terms and conditions. 
As an illustration, for the `llama2` model, you can use this link:
`https://huggingface.co/meta-llama/Llama-2-7b-hf <https://huggingface.co/meta-llama/Llama-2-7b-hf>`_.

Set up credentials to access huggingface
----------------------------------------

Your credential to access huggingface can be found online at `https://huggingface.co/settings/tokens <https://huggingface.co/settings/tokens>`_.

You can set the token as an environmental variable, with ``export HUGGING_FACE_HUB_TOKEN=your_token_here``.


Incompatibility Between NVIDIA Driver and PyTorch Version
=========================================================

If you are using a NVIDIA GPU, you may face the following error:

.. code-block:: text

   UserWarning: CUDA initialization: The NVIDIA driver on your system is too old
   (found version 10010). Please update your GPU driver by downloading and installi
   ng a new version from the URL: http://www.nvidia.com/Download/index.aspx Alterna
   tively, go to: https://pytorch.org to install a PyTorch version that has been co
   mpiled with your version of the CUDA driver. (Triggered internally at  ..\c10\cu
   da\CUDAFunctions.cpp:112.)

This typically indicates that your CUDA driver version is not compatible with the PyTorch version you are using.

Go to `https://pytorch.org <https://pytorch.org>`_ to install a PyTorch version that has been compiled with your
version of the CUDA driver. **Do not install a cuda version smaller than 11.8, preferably between 11.8 and 12.1.**

Say if your CUDA driver version is 11.8, then you can install PyTorch with the following command:

.. code-block:: python

   pip install torch==2.0.1+cu118


Xinference service cannot be accessed from external systems through ``<IP>:9997``
=================================================================================

Use ``-H 0.0.0.0`` parameter in when starting Xinference:

.. code:: bash

   xinference-local -H 0.0.0.0

Then Xinference service will listen on all network interfaces (not limited to ``127.0.0.1`` or ``localhost``).

If you are using the :ref:`using_docker_image`, please add ``-p <PORT>:9997``
during the docker run command, then access is available through ``<IP>:<PORT>`` of
the local machine.

Launching a built-in model takes a long time, and sometimes the model fails to download
=======================================================================================

Xinference by default uses HuggingFace as the source for models. If your
machines are in Mainland China, there might be accessibility issues when
using built-in models.

To address this, add environment variable ``XINFERENCE_MODEL_SRC=modelscope`` when starting
the Xinference to change the model source to ModelScope, which is optimized
for Mainland China.

If you’re starting Xinference with Docker, include ``-e XINFERENCE_MODEL_SRC=modelscope``
during the docker run command.

When using the official Docker image, RayWorkerVllm died due to OOM, causing the model to fail to load
=======================================================================================================

Docker's ``--shm-size`` parameter is used to set the size of shared memory. 
The default size of shared memory (/dev/shm) is 64MB, which may be too small for vLLM backend.


You can increase its size by setting the ``--shm-size`` parameter as follows:

.. code:: bash

   docker run --shm-size=128g ...


Missing ``model_engine`` parameter when launching LLM models
============================================================

Since version ``v0.11.0``, launching LLM models requires an additional ``model_engine`` parameter.
For specific information, please refer to :ref:`here <about_model_engine>`.

Resolving MKL Threading Layer Conflicts
========================================

When starting the Xinference server, you may encounter the error: ``ValueError: Model architectures ['Qwen2ForCausalLM'] failed to be inspected. Please check the logs for more details.``

The underlying cause shown in the logs is:

.. code-block:: text

   Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp-a34b3233.so.1 library.
   Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.

This typically occurs when NumPy was installed via conda. Conda's NumPy is built with Intel MKL optimizations, which conflicts with the GNU OpenMP library (libgomp) already loaded in your environment.

Solution 1: Override the Threading Layer
-----------------------------------------

Force Intel's Math Kernel Library to use GNU's OpenMP implementation:

.. code-block:: bash

   MKL_THREADING_LAYER=GNU xinference-local

Solution 2: Reinstall NumPy with pip
-------------------------------------

Uninstall conda's NumPy and reinstall using pip:

.. code-block:: bash

   pip uninstall -y numpy && pip install numpy
   #Or just --force-reinstall
   pip install --force-reinstall numpy

Related Note: vLLM and PyTorch
-------------------------------

If you're using vLLM, avoid installing PyTorch with conda. Refer to the official vLLM installation guide for GPU-specific instructions: https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html

Configuring PyPI Mirrors to Speed Up Package Installation
==========================================================

If you're in Mainland China, using a PyPI mirror can significantly speed up package installation. Here are some commonly used mirrors:

- Tsinghua University: ``https://pypi.tuna.tsinghua.edu.cn/simple``
- Alibaba Cloud: ``https://mirrors.aliyun.com/pypi/simple/``
- Tencent Cloud: ``https://mirrors.cloud.tencent.com/pypi/simple``

However, be aware that some packages may not be available on certain mirrors. For example, if you're installing ``xinference[audio]`` using only the Aliyun mirror, the installation may fail.

This happens because ``num2words``, a dependency used by ``MeloTTS``, is not available on the Aliyun mirror. As a result, ``pip install xinference[audio]`` will resolve to older versions like ``xinference==1.2.0`` and ``xoscar==0.8.0`` (as of Oct 27, 2025).

These older versions are incompatible and will produce the error: ``MainActorPool.append_sub_pool() got an unexpected keyword argument 'start_method'``

.. code-block:: bash

   curl -s https://mirrors.aliyun.com/pypi/simple/num2words/ | grep -i "num2words"
   # Returns NOTHING! But it works on Tsinghua or Tencent mirrors.
   # uv pip install "xinference[audio]" will then install the following packages (as of Oct 27, 2025):
   + x-transformers==2.10.2
   + xinference==1.2.0
   + xoscar==0.8.0

To avoid this issue when installing the xinference audio package, use multiple mirrors:

.. code-block:: bash

   uv pip install xinference[audio] --index-url https://mirrors.aliyun.com/pypi/simple --extra-index-url https://pypi.tuna.tsinghua.edu.cn/simple

   # Optional: Set this globally in your uv config
   mkdir -p ~/.config/uv
   cat >> ~/.config/uv/uv.toml << EOF
   index-url = "https://mirrors.aliyun.com/pypi/simple"
   extra-index-url = ["https://pypi.tuna.tsinghua.edu.cn/simple"]
   EOF

Installing Xinference 1.12.0 with uv Fails (As of November 2025)
=================================================================

**Note:** This is a temporary issue due to the current package ecosystem and uv prioritizing **higher versions for direct dependencies** over **indirect dependencies**.

Symptom
-------

When installing xinference 1.12.0 as of November 2025 using ``uv pip install xinference``, you may encounter an issue where very old package versions are installed, particularly:

- ``transformers==4.12.2`` (from 2021)
- ``tokenizers==0.10.3`` (from 2021)  
- ``huggingface-hub==1.0.1``

Then uv fails with "Failed to build `tokenizers==0.10.3`"

Root Cause
----------

This occurs because uv prioritizes **higher versions for direct dependencies** over **indirect dependencies**:

1. xinference 1.12.0 specifies ``huggingface-hub>=0.19.4`` as a **direct dependency** (no upper bound)
2. uv selects the latest: ``huggingface-hub==1.0.1`` as of November 06 2025
3. However, ``transformers<=4.57.3`` (an **indirect dependency** via ``peft``) requires ``huggingface-hub<1.0``
4. To resolve the conflict, uv keeps the direct dependency at 1.0.1 and downgrades the indirect dependency ``transformers`` to ancient version 4.12.2

**This is by design in uv**: it prioritizes what you explicitly ask for (direct dependencies) over transitive dependencies. Refer to https://github.com/astral-sh/uv/issues/16601

**Update:** The latest transformers 4.57.3 (as in 2026.01.05) still requires ``huggingface-hub<1.0``.

Solutions
---------

**Solution 1: Pre-constrain huggingface-hub (Recommended)**

Explicitly constrain ``huggingface-hub`` to a compatible version range:

.. code-block:: bash

   uv pip install "huggingface-hub>=0.34.0,<1.0" xinference

This forces uv to select a ``huggingface-hub`` version that's compatible with modern ``transformers``.

**Solution 2: Make transformers a direct dependency**

By specifying ``transformers`` explicitly, it becomes a direct dependency and uv will prefer higher versions:

.. code-block:: bash

   uv pip install transformers xinference

**Solution 3: Use pip**

Or just resort to using ``pip install xinference`` which will resolve to the following versions

- ``transformers==4.57.1``
- ``huggingface-hub==0.36.0``
- ``tokenizers==0.22.1``  

vLLM + Torch + Xinference Compatibility Issue (Segmentation Fault)
===================================================================

Symptom
-------

If you have **vLLM < 0.12.0** installed and upgrade xinference (particularly using ``uv pip install -U xinference``), xinference may fail to start with a segmentation fault:

.. code-block:: text

   root@server:/home# xinference-local --host 0.0.0.0 --port 9997
   INFO 12-30 17:35:37 [__init__.py:216] Automatically detected platform cuda.
   Aborted (core dumped)

Root Cause
----------

This issue has three contributing factors:

1. **Binary Incompatibility**: vLLM versions before 0.12.0 were compiled against PyTorch 2.8.0. These versions are incompatible with PyTorch 2.9. Reference: `vLLM v0.12.0 Release Notes <https://github.com/vllm-project/vllm/releases/tag/v0.12.0>`_

2. **Xinference's Unbounded Torch Dependency**: Xinference's ``pyproject.toml`` does not specify an upper bound for PyTorch:

   .. code-block:: toml

      [project]
      dependencies = [
          "torch",                 # No version constraint!
      ]

   This allows package managers to upgrade PyTorch to incompatible versions.

3. **Different Package Manager Behaviors**:

   - **pip**: Conservative - only upgrades the specified package unless dependencies are incompatible
   - **uv with -U flag**: Aggressive - re-resolves ALL dependencies and picks latest versions


Therefore before you're ready to upgrade your entire stack and just want to upgrade xinference, use either:

- ``pip install -U xinference`` (keeps PyTorch unchanged, only upgrades xinference)
- ``uv pip install "xinference==1.16.0"`` (without -U flag, only upgrades xinference too)

---

### Doc Source Getting Started Using Docker Compose (`doc/source/getting_started/using_docker_compose.rst`)

.. _using_docker_compose:

=========================
Docker Compose Deployment
=========================

Xinference ships an official Docker Compose setup for standalone deployment, located at
`xinference/deploy/docker <https://github.com/xorbitsai/inference/tree/main/xinference/deploy/docker>`_.
It supports online hosts as well as fully offline / air-gapped environments, where an optional
private PyPI server is started alongside Xinference to serve the packages installed at
model-launch time.

Prerequisites
=============
* Docker Compose **v2.24.4 or above** (required by the ``env_file: required: false`` and ``!reset`` features used in the compose files).
* For GPU deployment: a host with NVIDIA GPUs, CUDA installed, and `NVIDIA Container Toolkit <https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html>`_. See :ref:`using_docker_image` for image requirements.
* Get the whole ``xinference/deploy/docker`` directory. The compose file bind-mounts ``pip.conf`` from the same directory, so downloading ``docker-compose.yml`` alone is not sufficient:

.. code-block:: bash

   git clone https://github.com/xorbitsai/inference.git
   cd inference/xinference/deploy/docker


Quick Start (GPU)
=================
Start Xinference with all GPUs of the host:

.. code-block:: bash

   docker compose up -d

Wait until the service is healthy, then verify:

.. code-block:: bash

   docker compose ps
   curl http://localhost:9997/status

The Web UI is served at ``http://localhost:9997``.

.. note::

   Since v3.0, authentication is enabled by default: the first visit to the
   Web UI asks you to create the initial admin account, and API calls require
   a login or API key afterwards. To run without authentication, add
   ``XINFERENCE_AUTH_ADVANCED=false`` under the ``environment:`` section of
   ``docker-compose.yml``. See :ref:`user_guide_auth_system`.


Configuration
=============
All settings are exposed as variables with sensible defaults and can be overridden through a
``.env`` file next to ``docker-compose.yml``. Copy the template and edit as needed:

.. code-block:: bash

   cp .env.example .env

Available variables:

* ``XINFERENCE_IMAGE``: image to run, defaults to ``xprobe/xinference:latest``. Pin a release tag such as ``xprobe/xinference:v<version>`` for production.
* ``XINFERENCE_PORT``: host port of the RESTful API / Web UI, defaults to ``9997``.
* ``XINFERENCE_MODEL_SRC``: model download source, ``auto`` (default, probes Hugging Face reachability and falls back to ModelScope), ``huggingface`` or ``modelscope``.
* ``XINFERENCE_SHM_SIZE``: shared memory size, defaults to ``8gb``. Increase for multi-GPU inference.
* ``XINFERENCE_LOG_LEVEL``: log level, defaults to ``info``.
* ``XOSCAR_AUTH_KEY``: optional shared actor transport secret, passed to the Xinference service and all supervisor/worker services in ``docker-compose-distributed.yml``. Set one generated value in ``.env`` or export it before running Compose; unset or empty keeps actor messages unauthenticated. Use images containing Xoscar 0.12.0 or later and follow :ref:`actor_transport_authentication` before enabling or rotating the key.
* ``XINFERENCE_HOME_DIR`` / ``XINFERENCE_HF_CACHE_DIR`` / ``XINFERENCE_MODELSCOPE_CACHE_DIR``: persistence locations. They default to named Docker volumes; point them at absolute host paths to reuse existing model caches, in the same way as described in :ref:`using_docker_image`.
* ``XINFERENCE_WHEELS_DIR`` / ``XINFERENCE_PYPISERVER_PORT`` / ``XINFERENCE_PYPISERVER_IMAGE``: offline profile settings, see below.

For other runtime options (authentication, OpenTelemetry, health-check tuning, ...), add the
corresponding variables under the ``environment:`` section of ``docker-compose.yml``.
See :ref:`environments` for the full list.


CPU-only Deployment
===================
On hosts without NVIDIA GPUs, apply the CPU override file, which switches the image to the
``-cpu`` variant and removes the GPU reservation:

.. code-block:: bash

   docker compose -f docker-compose.yml -f docker-compose.cpu.yml up -d


Health Check and Restart Policy
===============================
The ``xinference`` service reports health through the ``/status`` endpoint and restarts
automatically unless explicitly stopped (``restart: unless-stopped``). ``docker compose ps``
shows the health state; orchestration on top of compose can rely on it.


.. _docker_compose_offline:

Offline / Air-gapped Deployment
===============================
By default Xinference installs the extra Python packages declared by a model at launch time
into a per-model virtual environment (controlled by ``XINFERENCE_ENABLE_VIRTUAL_ENV``,
see :ref:`environments`). On a host without Internet access these installs would fail.

The ``offline`` compose profile solves this by starting a private PyPI server next to
Xinference. Its image, ``xprobe/xinference-pypiserver``, ships the index-compatible wheels
the runtime may install into per-model virtual environments — including the ``vllm`` /
``sglang`` CUDA stacks — so no wheel preparation is needed for supported models. The offline
configuration points every runtime ``pip`` / ``uv`` invocation inside the Xinference container
at it.

.. note::

   The prebuilt mirror's GPU stack targets CUDA 13.0. This does not remove the runtime's
   existing online support for CUDA 12.8/12.9, but those stacks are not included in this
   mirror image.

Step 1: Transfer the Docker images
----------------------------------
Transfer the Docker images to the offline host (``docker save`` / ``docker load``): the
Xinference image and the mirror image. Pin both to the **same release tag** so the mirror
contents match that release's model specs and engine dependency lists, and record the pins
in ``.env``:

.. code-block:: bash

   XINFERENCE_IMAGE=xprobe/xinference:v2.9.0
   XINFERENCE_PYPISERVER_IMAGE=xprobe/xinference-pypiserver:v2.9.0

Step 2: Enable the offline configuration
----------------------------------------
.. code-block:: bash

   cp offline.env.example offline.env

Then open ``pip.conf`` and uncomment the three lines of the offline block:

.. code-block:: ini

   [global]
   index-url = http://xinference-pypiserver:8080/simple
   extra-index-url = http://xinference-pypiserver:8080/simple
   trusted-host = xinference-pypiserver

.. note::

   All three pieces are required because they cover different code paths. ``pip.conf`` feeds
   Xinference's pip-config inheritance, which passes the private index explicitly to the
   per-model virtual-env installer; the ``UV_*`` variables in ``offline.env`` cover ``uv``
   invocations that do not carry index flags (such as the dependency-resolution dry-run);
   ``XINFERENCE_VIRTUAL_ENV_OFFLINE_INSTALL=1`` enables direct-wheel rewriting only for this
   self-contained mirror. A normal pip mirror configured by an online user does not enable
   that behavior.

Step 3: Start with the offline profile
--------------------------------------
.. code-block:: bash

   docker compose --profile offline up -d

Or set ``COMPOSE_PROFILES=offline`` in ``.env`` to omit the ``--profile`` flag. Combine with the
CPU override if needed:

.. code-block:: bash

   docker compose --profile offline -f docker-compose.yml -f docker-compose.cpu.yml up -d

The private index is also published on the host (default port ``8080``), so other machines on
the same network can reuse it with ``pip install -i http://<host>:8080/simple ...``.

.. note::

   When launching models with the **vLLM** or **SGLang** engines, Xinference by default resolves
   some dependencies from hardcoded public indexes (``wheels.vllm.ai``,
   ``download.pytorch.org``) and from direct wheel URLs (``sgl_kernel``). The offline
   ``pip.conf`` above overrides both: the private index replaces the public indexes, and
   direct wheel-URL requirements are resolved from it as ``name==version``. The baked mirror
   already carries these CUDA wheels. Alternatively, set ``XINFERENCE_ENABLE_VIRTUAL_ENV=0``
   in ``offline.env`` to skip runtime installs entirely and rely on the packages baked into
   the Xinference image.

   For the **llama.cpp** engine, the mirror carries the CPU build of ``xllamacpp`` from PyPI.
   Its GPU wheels live on a separate CUDA-specific index that is unavailable offline, so a GPU
   host falls back to the CPU build and logs a warning. To retain llama.cpp GPU acceleration,
   preinstall the matching ``xllamacpp`` GPU wheel in a custom runtime image.

.. warning::

   Model specifications containing ``git+`` or other non-wheel direct references cannot be
   represented faithfully by a Python simple index. In explicit offline-install mode,
   Xinference rejects them before attempting network egress and reports the offending
   requirement. The current built-ins in this category include the Transformers path of
   HunyuanOCR, MiniCPM-V-4.6, and MiniCPM-V-4.6-Thinking, plus FLUX.2-klein. Preinstall the
   required source revision in a custom image or replace it with an index-resolvable package
   before using these models air-gapped. The FlashInfer AOT repair fetched from its public
   wheel index is also skipped in explicit offline mode; Blackwell deployments that require it
   should bake those packages into a custom image.

Bring your own wheels (optional)
--------------------------------
To serve a self-curated wheel directory instead of the baked mirror — for example a small
subset for specific models — add the ``docker-compose.byo-wheels.yml`` override, which swaps
the image for the stock ``pypiserver/pypiserver:v2.3.2`` and mounts ``./wheels``:

Set ``XINFERENCE_BYO_PYPISERVER_IMAGE`` if that stock image has been mirrored into a private
registry. This setting is intentionally independent of ``XINFERENCE_PYPISERVER_IMAGE``, which
selects the prebuilt Xinference mirror in the normal offline profile.

.. code-block:: bash

   python3 -m pip download \
      --dest ./wheels \
      --only-binary=:all: \
      --python-version 312 \
      --platform manylinux2014_x86_64 \
      'transformers>=4.53.3' accelerate

   chmod -R a+rwX ./wheels   # pypiserver runs as UID 9898
   docker compose --profile offline \
      -f docker-compose.yml -f docker-compose.byo-wheels.yml up -d

Offline model weights
---------------------
The offline configuration sets ``HF_HUB_OFFLINE=1`` and ``TRANSFORMERS_OFFLINE=1``, so model
weights are read from the local cache only. Populate the cache before going offline, either by
pointing the cache variables in ``.env`` at host directories that already contain the models,
or by launching each model once on a connected host and copying the volumes. Models can also be
loaded from arbitrary local paths by registering custom models or passing ``--model-path``.

Enforced network isolation (optional)
-------------------------------------
The offline configuration redirects every download to local sources, but by itself it does not
prevent the containers from reaching the Internet if the host happens to have connectivity. To
*guarantee* isolation at the network level, add the air-gap override, which moves Xinference and
the private PyPI server onto an ``internal`` Docker network with no external routing:

.. code-block:: bash

   docker compose --profile offline \
      -f docker-compose.yml -f docker-compose.airgap.yml up -d

Because Docker does not publish ports of internal-only networks, the override adds a minimal
TCP gateway (``alpine/socat``) that forwards the API port from the host into the isolated
network. The gateway only relays inbound traffic to ``xinference:9997``; the containers cannot
use it as an egress path. Remember to transfer the ``alpine/socat`` image to the offline host
along with the others.

.. note::

   In this mode the private PyPI server is not published on the host; it is only reachable
   from containers inside the isolated network. Verify the isolation from within the
   container — external requests must fail while the private index stays reachable:

   .. code-block:: bash

      docker compose -f docker-compose.yml -f docker-compose.airgap.yml exec xinference \
         curl -s -m 5 https://pypi.org -o /dev/null || echo "external access blocked"
      docker compose -f docker-compose.yml -f docker-compose.airgap.yml exec xinference \
         curl -s http://xinference-pypiserver:8080/health

Smoke test
----------
.. code-block:: bash

   # The private index serves your wheels:
   curl http://localhost:8080/simple/

   # Inside the container, installs resolve against the private index only:
   docker compose exec xinference python3 -m pip config list
   docker compose exec xinference uv pip install --dry-run --python /usr/bin/python3 <some-package-in-wheels>

Then launch a model and watch ``docker compose logs -f xinference`` — package installation logs
should reference ``http://xinference-pypiserver:8080/simple``.

---

### Doc Source Getting Started Using Docker Image (`doc/source/getting_started/using_docker_image.rst`)

.. _using_docker_image:

=======================
Xinference Docker Image
=======================

Xinference provides official images for use on Dockerhub.

.. versionchanged:: v2.0

   Starting from **Xinference v2.0**, to use the CUDA version of the image, the minimum CUDA version must be **CUDA 12.9**.

.. versionchanged:: v3.0.0

   Starting from **Xinference v3.0.0**, the GPU image is a *slim* image based on
   ``nvidia/cuda:13.0.2-devel-ubuntu22.04``. It no longer pre-installs inference
   engines such as vLLM or SGLang; engines are installed automatically into a
   per-model virtual environment at model launch time (see
   ``XINFERENCE_ENABLE_VIRTUAL_ENV`` in :ref:`environments`). This requires access
   to PyPI at model launch — for offline / air-gapped hosts, use the private PyPI
   mirror of the Docker Compose offline profile, see :ref:`using_docker_compose`.
   The minimum supported CUDA version of the image is **CUDA 13.0**.

Prerequisites
=============
* The image can only run in an environment with GPUs and CUDA installed, because Xinference in the image relies on Nvidia GPUs for acceleration.
* CUDA must be successfully installed on the host machine. This can be determined by whether you can successfully execute the ``nvidia-smi`` command.
* The CUDA version in the docker image is ``13.0``; the CUDA version on the host machine should be ``13.0`` or above, and the NVIDIA driver version should be ``580`` or above.
* Ensure `NVIDIA Container Toolkit <https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html>`_ installed.


Docker Image
============
The official image of Xinference is available on DockerHub in the repository ``xprobe/xinference``.
Available tags include:

* ``nightly-main``: This image is built daily from the `GitHub main branch <https://github.com/xorbitsai/inference>`_ and generally does not guarantee stability.
* ``v<release version>``: This image is built each time a Xinference release version is published, and it is typically more stable.
* ``latest``: This image is built with the latest Xinference release version.
* For CPU version, add ``-cpu`` suffix, e.g. ``nightly-main-cpu``.

What is inside the GPU image
============================
The GPU image ships Python 3.12, the CUDA 13.0 toolkit, a shared CUDA build of
PyTorch, the Transformers engine (``transformers`` / ``accelerate``) and
Xinference itself with the Web UI. Other inference engines (vLLM, SGLang,
llama.cpp, ...) are **not** pre-installed: when a model is launched, Xinference
creates a virtual environment for it and installs the engine packages declared
by the model spec. The first launch of a model with a given engine therefore
downloads packages from PyPI (or from a private mirror in offline deployments)
and takes longer; subsequent launches reuse the virtual environment.

.. note::

   For offline deployments that install engine dependencies into per-model virtual
   environments at model launch time, use the Docker Compose ``offline`` profile
   to provide a private PyPI mirror. See
   :ref:`Docker Compose offline deployment <docker_compose_offline>` for image
   preparation and dependency source configuration.


Dockerfile for custom build
===========================
If you need to build the Xinference image according to your own requirements, the source code for the Dockerfile is located at `xinference/deploy/docker/Dockerfile <https://github.com/xorbitsai/inference/tree/main/xinference/deploy/docker/Dockerfile>`_ for reference.
Please make sure to be in the top-level directory of Xinference when using this Dockerfile. For example:

.. code-block:: bash

   git clone https://github.com/xorbitsai/inference.git
   cd inference
   docker build --progress=plain -t test -f xinference/deploy/docker/Dockerfile .


Image usage
===========
You can start Xinference in the container like this, simultaneously mapping port 9997 in the container to port 9998 on the host, enabling debug logging, and downloading models from modelscope.

.. code-block:: bash

   docker run -e XINFERENCE_MODEL_SRC=modelscope -p 9998:9997 --gpus all xprobe/xinference:v<your_version> xinference-local -H 0.0.0.0 --log-level debug


.. warning::
    * The option ``--gpus`` is essential and cannot be omitted, because as mentioned earlier, the image requires the host machine to have a GPU. Otherwise, errors will occur.
    * The ``-H 0.0.0.0`` parameter after the ``xinference-local`` command cannot be omitted. Otherwise, the host machine may not be able to access the port inside the container.
    * You can add multiple ``-e`` options to introduce multiple environment variables.


Certainly, if you prefer, you can also manually enter the docker container and start Xinference in any desired way.

.. note::

   For multiple GPUs, make sure to set the shared memory size, for example: `docker run --shm-size=128g ...`


Mount your volume for loading and saving models
===============================================
The image does not contain any model files by default, and it downloads the models into the container.
Typically, you would need to mount a directory on the host machine to the docker container, so that Xinference can download the models onto it, allowing for reuse.
In this case, you need to specify a volume when running the Docker image and configure environment variables for Xinference:

.. code-block:: bash

   docker run -v </on/your/host>:</on/the/container> -e XINFERENCE_HOME=</on/the/container> -p 9998:9997 --gpus all xprobe/xinference:v<your_version> xinference-local -H 0.0.0.0


The principle behind the above command is to mount the specified directory from the host machine into the container, and then set the ``XINFERENCE_HOME`` environment variable to point to that directory inside the container.
This way, all downloaded model files will be stored in the directory you specified on the host machine.
You don't have to worry about losing them when the Docker container stops, and the next time you run it, you can directly use the existing models without the need for repetitive downloads.

If you downloaded the model using the default path on the host machine, and since the xinference cache directory
stores the model using symbolic links, you need to mount the directory where the original file is located into the container as well.
For example, if you are using HuggingFace and Modelscope as model hub, you would need to mount the corresponding
directories into the container. Generally, the cache directories for HuggingFace and Modelscope are located
at <home_path>/.cache/huggingface and <home_path>/.cache/modelscope. The command would be like:

.. code-block:: bash

   docker run \
     -v </your/home/path>/.xinference:/root/.xinference \
     -v </your/home/path>/.cache/huggingface:/root/.cache/huggingface \
     -v </your/home/path>/.cache/modelscope:/root/.cache/modelscope \
     -p 9997:9997 \
     --gpus all \
     xprobe/xinference:v<your_version> \
     xinference-local -H 0.0.0.0

---

### Doc Source Getting Started Using Kubernetes (`doc/source/getting_started/using_kubernetes.rst`)

.. _using_kubernetes:

########################
Xinference on Kubernetes
########################

************
Helm Support
************
Xinference provides a method for installation in a Kubernetes cluster via ``Helm`` .


Prerequisites
=============
* You have a fully functional Kubernetes cluster.
* Enable GPU support in Kubernetes, refer to `here <https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/>`_.
* ``Helm`` is correctly installed.


Steps
=====
#. Add xinference helm repo.

    .. code-block:: bash

      helm repo add xinference https://xorbitsai.github.io/xinference-helm-charts

#. Update xinference helm repo indexes and query versions.

    .. code-block:: bash

      helm repo update xinference
      helm search repo xinference/xinference --devel --versions

#. Install

    .. code-block:: bash

      helm install xinference xinference/xinference -n xinference --version <helm_charts_version>


Customized Installation
=======================
The installation method mentioned above sets up a Xinference cluster similar to a single-machine setup,
with only one worker and all startup parameters at their default values.
However, this is usually not the desired setup.

Below are some common custom installation configurations.

#. I need to download models from ``ModelScope``.

    .. code-block:: bash

      helm install xinference xinference/xinference -n xinference --version <helm_charts_version> --set config.model_src="modelscope"

#. I want to use cpu image of xinference (or use any other version of xinference images).

    .. code-block:: bash

      helm install xinference xinference/xinference -n xinference --version <helm_charts_version> --set config.xinference_image="<xinference_docker_image>"

#. I want to have 4 Xinference workers, with each worker managing 4 GPUs.

    .. code-block:: bash

      helm install xinference xinference/xinference -n xinference --version <helm_charts_version> --set config.worker_num=4 --set config.gpu_per_worker="4"

The above installation method is based on Helm ``--set`` option.
For more complex custom installations, such as multiple workers with shared storage,
it is highly recommended to use your own ``values.yaml`` file with Helm ``-f`` option for installation.

The default ``values.yaml`` file is located `here <https://github.com/xorbitsai/xinference-helm-charts/blob/main/charts/xinference/values.yaml>`_.
Some examples can be found `here <https://github.com/xorbitsai/xinference-helm-charts/tree/main/examples>`_.


******************
KubeBlocks Support
******************
You can also install Xinference in Kubernetes using the third-party ``KubeBlocks``.
This method is not maintained by Xinference and does not guarantee timely updates or availability.
Please refer to the documentation at `here <https://kubeblocks.io/docs/preview/user_docs/kubeblocks-for-xinference/manage-xinference>`_.

---

### Doc Source Getting Started Using Xinference (`doc/source/getting_started/using_xinference.rst`)

.. _using_xinference:

================
Using Xinference
================


Run Xinference Locally
======================

Let's start by running Xinference on a local machine and running a classic LLM model: ``qwen2.5-instruct``.

After this quickstart, you will move on to learning how to deploy Xinference in a cluster environment.

Start Local Server
------------------

First, please ensure that you have installed Xinference according to the instructions provided :ref:`here <installation>`.
To start a local instance of Xinference, run the following command:

.. tabs::

  .. tab:: shell

    .. code-block:: bash

        xinference-local --host 0.0.0.0 --port 9997

  .. tab:: output

    .. code-block:: bash

      INFO     Xinference supervisor 0.0.0.0:64570 started
      INFO     Xinference worker 0.0.0.0:64570 started
      INFO     Starting Xinference at endpoint: http://0.0.0.0:9997
      INFO     Uvicorn running on http://0.0.0.0:9997 (Press CTRL+C to quit)

.. note::
  By default, Xinference uses ``<HOME>/.xinference`` as home path to store necessary files such as logs and models,
  where ``<HOME>`` is the home path of current user.

  You can change this directory by configuring the environment variable ``XINFERENCE_HOME``.
  For example:

  .. code-block:: bash

    XINFERENCE_HOME=/tmp/xinference xinference-local --host 0.0.0.0 --port 9997

Congrats! You now have Xinference running on your local machine. Once Xinference is running, there are multiple ways
we can try it: via the web UI, via cURL, via the command line, or via the Xinference's python client.

You can visit the web UI at `http://127.0.0.1:9997 <http://127.0.0.1:9997>`_ and visit `http://127.0.0.1:9997/docs <http://127.0.0.1:9997/docs>`_
to inspect the API docs.

.. note::

   The web UI is a Next.js application bundled into the ``xinference`` package as a static
   export and served by the backend itself, so no Node.js runtime is required at runtime.
   When developing Xinference from a source checkout, build it once with
   ``cd frontend && npm ci && npm run build``, or work on the frontend live with
   ``npm run dev`` (see the "Creating a development environment" page).

You can install the Xinference command line tool and Python client using the following command:

.. code-block:: bash

   pip install xinference

The command line tool is ``xinference``. You can list the commands that can be used by running:

.. tabs::

  .. tab:: shell

    .. code-block:: bash

      xinference --help

  .. tab:: output

    .. code-block:: bash

      Usage: xinference [OPTIONS] COMMAND [ARGS]...

      Options:
        -v, --version       Show the version and exit.
        --log-level TEXT
        -H, --host TEXT
        -p, --port INTEGER
        --help              Show this message and exit.

      Commands:
        cached
        cal-model-mem
        chat
        engine
        generate
        launch
        list
        login
        register
        registrations
        remove-cache
        stop-cluster
        terminate
        unregister
        vllm-models


You can install the Xinference Python client with minimal dependencies using the following command.
Please ensure that the version of the client matches the version of the Xinference server.

.. code-block:: bash

   pip install xinference-client==${SERVER_VERSION}

.. _about_model_engine:

About Model Engine
------------------
Since ``v0.11.0`` , before launching the LLM model, you need to specify the inference engine you want to run.
Currently, xinference supports the following inference engines:

* ``vllm``
* ``sglang``
* ``llama.cpp``
* ``transformers``
* ``MLX``

About the details of these inference engine, please refer to :ref:`here <inference_backend>`.

.. tip::

   .. versionadded:: v3.5.0

   Use **Recommend configuration** in the Web UI's model deployment dialog to
   fill in a suggested engine, format, size, and quantization. See
   :ref:`launch recommendations <launch_recommendations>` for details.

Note that when launching a LLM model, the ``model_format`` and ``quantization`` of the model you want to launch
is closely related to the inference engine.

You can use ``xinference engine`` command to query the combination of parameters of the model you want to launch.
This will demonstrate under what conditions a model can run on which inference engines.

For LLM, embedding and rerank models, engine discovery combines results from all registered workers.
Engine names and parameter combinations retain their first-seen order, with duplicate combinations
removed; an available engine takes precedence over another worker's unavailability message. Empty
worker results are skipped, and local discovery is used if all results are empty. A worker query
error fails the request instead of returning an incomplete capability list.

For example:

#. I would like to query about which inference engines the ``qwen-chat`` model can run on, and what are their respective parameters.

.. code-block:: bash

    xinference engine -e <xinference_endpoint> --model-name qwen-chat

#. I want to run ``qwen-chat`` with ``VLLM`` as the inference engine, but I don't know how to configure the other parameters.

.. code-block:: bash

    xinference engine -e <xinference_endpoint> --model-name qwen-chat --model-engine vllm

#. I want to launch the ``qwen-chat`` model in the ``GGUF`` format, and I need to know how to configure the remaining parameters.

.. code-block:: bash

    xinference engine -e <xinference_endpoint> --model-name qwen-chat -f ggufv2


In summary, compared to previous versions, when launching LLM models,
you need to additionally pass the ``model_engine`` parameter.
You can retrieve information about the supported inference engines and their related parameter combinations
through the ``xinference engine`` command.

.. note::

    Here are some recommendations on when to use which engine:

    - **Linux**

       - When possible, prioritize using **vLLM** or **SGLang** for better performance.
       - If resources are limited, consider using **llama.cpp**, as it offers more quantization options.
       - For other cases, consider using **Transformers**, which supports nearly all models.

    - **Windows**

       - It is recommended to use **WSL**, and in this case, follow the same choices as Linux.
       - Otherwise, prefer **llama.cpp**, and for unsupported models, opt for **Transformers**.

    - **Mac**

       - If supported by the model, use the **MLX engine**, as it delivers the best performance.
       - For other cases, prefer **llama.cpp**, and for unsupported models, choose **Transformers**.


Run qwen2.5-instruct
--------------------

Let's start by running a built-in model: ``qwen2.5-instruct``. When you start a model for the first time, Xinference will
download the model parameters from HuggingFace, which might take a few minutes depending on the size of the model weights.
We cache the model files locally, so there's no need to redownload them for subsequent starts.

.. note::
  Xinference also allows you to download models from other sites. You can do this by setting an environment variable
  when launching Xinference. For example, if you want to download models from `modelscope <https://modelscope.cn>`_,
  do the following:

  .. code-block:: bash

    XINFERENCE_MODEL_SRC=modelscope xinference-local --host 0.0.0.0 --port 9997

We can specify the model's UID using the ``--model-uid`` or ``-u`` flag. If not specified, Xinference will generate a unique ID.
The default unique ID will be identical to the model name.

.. tabs::

  .. code-tab:: bash shell

    xinference launch --model-engine <inference_engine> -n qwen2.5-instruct -s 0_5 -f pytorch

  .. code-tab:: bash cURL

    curl -X 'POST' \
      'http://127.0.0.1:9997/v1/models' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
      "model_engine": "<inference_engine>",
      "model_name": "qwen2.5-instruct",
      "model_format": "pytorch",
      "size_in_billions": "0_5"
    }'

  .. code-tab:: python

    from xinference.client import RESTfulClient
    client = RESTfulClient("http://127.0.0.1:9997")
    model_uid = client.launch_model(
      model_engine="<inference_engine>",
      model_name="qwen2.5-instruct",
      model_format="pytorch",
      size_in_billions="0_5"
    )
    print('Model uid: ' + model_uid)

  .. code-tab:: bash output

    Model uid: qwen2.5-instruct

.. note::
  For some engines, such as vllm, users need to specify the engine-related parameters when
  running models. In this case, you can directly specify the parameter name and value in the
  command line, for example:

  .. code-block:: bash

    xinference launch --model-engine vllm -n qwen2.5-instruct -s 0_5 -f pytorch --gpu_memory_utilization 0.9

  `gpu_memory_utilization=0.9` will pass to vllm when launching model.

.. note::
  For more tips on model launching, refer to :ref:`launch`.

Congrats! You now have ``qwen2.5-instruct`` running by Xinference. Once the model is running, we can try it out either via cURL,
or via Xinference's python client:

.. tabs::

  .. code-tab:: bash cURL

    curl -X 'POST' \
      'http://127.0.0.1:9997/v1/chat/completions' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "qwen2.5-instruct",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "What is the largest animal?"
            }
        ]
      }'

  .. code-tab:: python

    from xinference.client import RESTfulClient
    client = RESTfulClient("http://127.0.0.1:9997")
    model = client.get_model("qwen2.5-instruct")
    model.chat(
        messages=[
            {"role": "user", "content": "Who won the world series in 2020?"}
        ]
    )

  .. code-tab:: json output

    {
      "id": "chatcmpl-8d76b65a-bad0-42ef-912d-4a0533d90d61",
      "model": "qwen2.5-instruct",
      "object": "chat.completion",
      "created": 1688919187,
      "choices": [
        {
          "index": 0,
          "message": {
            "role": "assistant",
            "content": "The largest animal that has been scientifically measured is the blue whale, which has a maximum length of around 23 meters (75 feet) for adult animals and can weigh up to 150,000 pounds (68,000 kg). However, it is important to note that this is just an estimate and that the largest animal known to science may be larger still. Some scientists believe that the largest animals may not have a clear \"size\" in the same way that humans do, as their size can vary depending on the environment and the stage of their life."
          },
          "finish_reason": "None"
        }
      ],
      "usage": {
        "prompt_tokens": -1,
        "completion_tokens": -1,
        "total_tokens": -1
      }
    }

Xinference provides OpenAI-compatible APIs for its supported models, so you can use Xinference as a local drop-in replacement for OpenAI APIs. For example:

.. code-block:: python

  from openai import OpenAI
  client = OpenAI(base_url="http://127.0.0.1:9997/v1", api_key="not used actually")

  response = client.chat.completions.create(
      model="qwen2.5-instruct",
      messages=[
          {"role": "system", "content": "You are a helpful assistant."},
          {"role": "user", "content": "What is the largest animal?"}
      ]
  )
  print(response)

The following OpenAI APIs are supported:

- Chat Completions: `https://platform.openai.com/docs/api-reference/chat <https://platform.openai.com/docs/api-reference/chat>`_

- Completions: `https://platform.openai.com/docs/api-reference/completions <https://platform.openai.com/docs/api-reference/completions>`_

- Embeddings: `https://platform.openai.com/docs/api-reference/embeddings <https://platform.openai.com/docs/api-reference/embeddings>`_

- Responses (stateless, used by Codex CLI): `https://platform.openai.com/docs/api-reference/responses <https://platform.openai.com/docs/api-reference/responses>`_. Refer to :ref:`responses client <openai_responses_client>` for more details.

Xinference also supports Anthropic API via base url ``http://127.0.0.1:9997/anthropic``, you can use Xinference in Claude Code and so forth.
Refer to :ref:`anthropic client <anthropic_client>` for more details.

Manage Models
-------------

In addition to launching models, Xinference offers various ways to manage the entire lifecycle of models.
You can manage models in Xinference through the command line, cURL, or Xinference's python client.

You can list all models of a certain type that are available to launch in Xinference:

.. tabs::

  .. code-tab:: bash shell

    xinference registrations -t LLM

  .. code-tab:: bash cURL

    curl http://127.0.0.1:9997/v1/model_registrations/LLM

  .. code-tab:: python

    from xinference.client import RESTfulClient
    client = RESTfulClient("http://127.0.0.1:9997")
    print(client.list_model_registrations(model_type='LLM'))

The following command gives you the currently running models in Xinference:

.. tabs::

  .. code-tab:: bash shell

    xinference list

  .. code-tab:: bash cURL

    curl http://127.0.0.1:9997/v1/models

  .. code-tab:: python

    from xinference.client import RESTfulClient
    client = RESTfulClient("http://127.0.0.1:9997")
    print(client.list_models())

When you no longer need a model that is currently running, you can remove it in the following way to free up the resources it occupies:

.. tabs::

  .. code-tab:: bash shell

    xinference terminate --model-uid "qwen2.5-instruct"

  .. code-tab:: bash cURL

    curl -X DELETE http://127.0.0.1:9997/v1/models/qwen2.5-instruct

  .. code-tab:: python

    from xinference.client import RESTfulClient
    client = RESTfulClient("http://127.0.0.1:9997")
    client.terminate_model(model_uid="qwen2.5-instruct")

.. _distributed_getting_started:

Deploy Xinference In a Cluster
==============================

To deploy Xinference in a cluster, you need to start a Xinference supervisor on one server and Xinference workers
on the other servers.

.. warning::

   Before starting the cluster, configure the same ``XOSCAR_AUTH_KEY`` on the
   supervisor and every worker as described in :ref:`actor_transport_authentication`.
   HTTP API authentication does not protect actor ports; an unset or empty key
   leaves the actor transport unauthenticated. Limit actor-port access to trusted
   cluster peers using your deployment's network controls.

First, make sure you have already installed Xinference on each of the servers according to the instructions
provided :ref:`here <installation>`. Then follow the steps below:

Start the Supervisor
--------------------
On the server where you want to run the Xinference supervisor, run the following command:

.. code-block:: bash

  xinference-supervisor -H "${supervisor_host}"

Replace ``${supervisor_host}`` with the actual host of your supervisor server.


You can the supervisor's web UI at `http://${supervisor_host}:9997/ui <http://${supervisor_host}:9997/ui>`_ and visit
`http://${supervisor_host}:9997/docs <http://${supervisor_host}:9997/docs>`_ to inspect the API docs.

Start the Workers
-----------------

On each of the other servers where you want to run Xinference workers, run the following command:

.. code-block:: bash

  xinference-worker -e "http://${supervisor_host}:9997" -H "${worker_host}"

.. note::
    Note that you must replace ``${worker_host}``  with the actual host of your worker server.

.. note::
  Note that if you need to interact with the Xinference in a cluster via the command line,
  you should include the ``-e`` or ``--endpoint`` flag to specify the supervisor server's endpoint. For example:

  .. code-block:: bash

      xinference launch -n qwen2.5-instruct -s 0_5 -f pytorch -e "http://${supervisor_host}:9997"

Using Xinference With Docker
=============================

To start Xinference in a Docker container, run the following command:

Run On Nvidia GPU Host
-----------------------

For cuda 12.4:

.. code-block:: bash

  docker run -e XINFERENCE_MODEL_SRC=modelscope -p 9998:9997 --gpus all xprobe/xinference:<your_version> xinference-local -H 0.0.0.0 --log-level debug

For cuda 12.8:

.. versionadded:: v1.8.1
  CUDA 12.8 version is experimental, welcome to give us feedbacks to help us to improve.

.. versionchanged:: v1.16.0
  CUDA 12.8 version is removed in v1.16.0 .

.. code-block:: bash

  docker run -e XINFERENCE_MODEL_SRC=modelscope -p 9998:9997 --gpus all xprobe/xinference:<your_version>-cu128 xinference-local -H 0.0.0.0 --log-level debug

For cuda 12.9:

.. versionadded:: v1.16.0
  CUDA 12.9 will become the default version when Xinference v2.0.0 released.

.. code-block:: bash

  docker run -e XINFERENCE_MODEL_SRC=modelscope -p 9998:9997 --gpus all xprobe/xinference:<your_version>-cu129 xinference-local -H 0.0.0.0 --log-level debug

Run On CPU Only Host
-----------------------

.. code-block:: bash

  docker run -e XINFERENCE_MODEL_SRC=modelscope -p 9998:9997 xprobe/xinference:<your_version>-cpu xinference-local -H 0.0.0.0 --log-level debug

Replace ``<your_version>`` with Xinference versions, e.g. ``v0.10.3``, ``latest`` can be used for the latest version.

For more docker usage, refer to :ref:`Using Docker Image <using_docker_image>`.


What's Next?
============

Congratulations on getting started with Xinference! To help you navigate and make the most out of this
powerful tool, here are some resources and guides:

* :ref:`How to Use Client APIs for Different Types of Models <user_guide_client_api>`

* :ref:`Choosing the Right Backends for Your Needs <user_guide_backends>`

---

### Doc Source Models Custom (`doc/source/models/custom.rst`)

.. _models_custom:

=============
Custom Models
=============
Xinference provides a flexible and comprehensive way to integrate, manage, and utilize custom models.


Directly launch an existing model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Since ``v0.14.0``, you can directly launch an existing model by passing ``model_path`` to the launch interface without downloading it.
This way requires that the model's ``model_family`` is among the built-in supported models,
and eliminates the hassle of registering the model.

For example:

.. tabs::

  .. code-tab:: bash shell

    xinference launch --model-path <model_file_path> --model-engine <engine> -n qwen1.5-chat

  .. code-tab:: bash cURL

    curl -X 'POST' \
      'http://127.0.0.1:9997/v1/models' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
      "model_engine": "<engine>",
      "model_name": "qwen1.5-chat",
      "model_path": "<model_file_path>"
    }'

  .. code-tab:: python

    from xinference.client import RESTfulClient
    client = RESTfulClient("http://127.0.0.1:9997")
    model_uid = client.launch_model(
      model_engine="",
      model_name="qwen1.5-chat",
      model_path="<model_file_path>"
    )
    print('Model uid: ' + model_uid)


The above example demonstrates how to directly launch a qwen1.5-chat model file without registering it.

For distributed scenarios, if your model file is on a specific worker,
you can directly launch it using the ``worker_ip`` and ``model_path`` parameters with the launch interface.

.. note::
   For CLI usage, prefer ``--model-path`` (kebab-case). ``--model_path`` is legacy-compatible but not recommended.

Define a custom model
~~~~~~~~~~~~~~~~~~~~~~~~~

Web UI: Automatic LLM Config Parsing
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. versionadded:: v2.0.0

When registering a custom LLM via the Web UI, Xinference can automatically parse the model
configuration and pre-fill key fields for you.

You only need to provide:

- **Model path / Model ID** (where the model lives, local path or hub ID)
- **Model Family**

After parsing, the UI can auto-populate fields such as:

- ``Context Length``
- ``Model_Languages``
- ``Model_Abilities``
- ``Model_Specs``

You can review and edit these fields before saving the custom model.

Define a custom model based on the following templates:

.. tabs::

  .. tab:: LLM

    .. code-block:: json

        {
            "version": 2,
            "context_length": 32768,
            "model_name": "custom-qwen-2.5",
            "model_lang": [
                "en",
                "zh"
            ],
            "model_ability": [
                "generate"
            ],
            "model_description": "This is a custom model description.",
            "model_family": "my-custom-qwen-2.5",
            "model_specs": [
                {
                    "model_format": "pytorch",
                    "model_size_in_billions": "0_5",
                    "quantization": "none",
                    "model_id": null,
                    "model_hub": "huggingface",
                    "model_uri": "file:///path/to/models--Qwen--Qwen2.5-0.5B",
                    "model_revision": null,
                    "activated_size_in_billions": null
                }
            ],
            "chat_template": null,
            "stop_token_ids": null,
            "stop": null,
            "reasoning_start_tag": null,
            "reasoning_end_tag": null,
            "cache_config": null,
            "virtualenv": {
                "packages": [],
                "inherit_pip_config": true,
                "index_url": null,
                "extra_index_url": null,
                "find_links": null,
                "trusted_host": null,
                "no_build_isolation": null
            },
            "is_builtin": false
        }

  .. tab:: embedding

    .. code-block:: json

      {
         "version": 2,
         "model_name": "my-bge-large-zh-v1.5",
         "dimensions": 1024,
         "max_tokens": 512,
         "language": [
             "zh"
         ],
         "model_specs": [
            {
                "model_format": "pytorch",
                "model_hub": "huggingface",
                "model_id": null,
                "model_uri": "file:///path/to/my-bge-large-zh-v1.5",
                "model_revision": null,
                "quantization": "none"
            }
         ],
         "cache_config": null,
         "virtualenv": {
            "packages": [],
            "inherit_pip_config": true,
            "index_url": null,
            "extra_index_url": null,
            "find_links": null,
            "trusted_host": null,
            "no_build_isolation": null
         },
         "is_builtin": false
      }

  .. tab:: Rerank

    .. code-block:: json

      {
        "version": 2,
        "model_name": "my-bge-reranker-base",
        "model_specs": [
            {
                "model_format": "pytorch",
                "model_hub": "huggingface",
                "model_id": null,
                "model_revision": null,
                "model_uri": "file:///path/to/my-bge-reranker-base",
                "quantization": "none"
            }
        ],
        "language": [
            "en",
            "zh"
        ],
        "type": "unknown",
        "max_tokens": 512,
        "virtualenv": {
            "packages": [],
            "inherit_pip_config": true,
            "index_url": null,
            "extra_index_url": null,
            "find_links": null,
            "trusted_host": null,
            "no_build_isolation": null
        },
        "is_builtin": false
      }

  .. tab:: image

    .. code-block:: json

      {
        "model_name": "my-qwen-image",
        "model_id": null,
        "model_revision": null,
        "model_hub": "huggingface",
        "cache_config": null,
        "version": 2,
        "model_family": "stable_diffusion",
        "model_ability": null,
        "controlnet": [],
        "default_model_config": {},
        "default_generate_config": {},
        "gguf_model_id": null,
        "gguf_quantizations": null,
        "gguf_model_file_name_template": null,
        "lightning_model_id": null,
        "lightning_versions": null,
        "lightning_model_file_name_template": null,
        "virtualenv": {
            "packages": [],
            "inherit_pip_config": true,
            "index_url": null,
            "extra_index_url": null,
            "find_links": null,
            "trusted_host": null,
            "no_build_isolation": null
        },
        "model_uri": "file:///path/to/my-qwen-image",
        "is_builtin": false
      }

  .. tab:: audio

    .. code-block:: json

      {
        "model_name": "my-ChatTTS",
        "model_id": null,
        "model_revision": null,
        "model_hub": "huggingface",
        "cache_config": null,
        "version": 2,
        "model_family": "ChatTTS",
        "multilingual": false,
        "language": null,
        "model_ability": [
            "text2audio"
        ],
        "default_model_config": null,
        "default_transcription_config": null,
        "engine": null,
        "virtualenv": {
            "packages": [],
            "inherit_pip_config": true,
            "index_url": null,
            "extra_index_url": null,
            "find_links": null,
            "trusted_host": null,
            "no_build_isolation": null
        },
        "model_uri": "file:///path/to/my-ChatTTS",
        "is_builtin": false
      }

  .. tab:: flexible

    .. code-block:: json

      {
        "model_name": "my-flexible-model",
        "model_id": null,
        "model_revision": null,
        "model_hub": "huggingface",
        "cache_config": null,
        "version": 2,
        "model_description": "This is a model description.",
        "model_uri": "file:///path/to/my-flexible-model",
        "launcher": "xinference.model.flexible.launchers.transformers",
        "launcher_args": "{}",
        "virtualenv": {
            "packages": [],
            "inherit_pip_config": true,
            "index_url": null,
            "extra_index_url": null,
            "find_links": null,
            "trusted_host": null,
            "no_build_isolation": null
        },
        "is_builtin": false
      }

* model_name: A string defining the name of the model. The name must start with a letter or a digit and can only contain letters, digits, underscores, or dashes.
* context_length: An optional integer that specifies the maximum context size the model was trained to accommodate, encompassing both the input and output lengths. If not defined, the default value is 2048 tokens (~1,500 words).
* dimensions: An interger defining the size of the vector output by the embedding model.
* max_tokens: An interger defining the maximum number of input tokens the embedding model can process in a single request.
* model_lang: A list of strings representing the supported languages for the model. Example: ["en"], which means that the model supports English.
* model_ability: A list of strings defining the abilities of the model. It could include options like "embed", "generate", and "chat". In this case, the model has the ability to "generate".
* model_family: A required string representing the family of the model you want to register. This parameter must not conflict with any builtin model names.
* model_specs: An array of objects defining the specifications of the model. These include:
   * model_format: A string that defines the model format, like "pytorch" or "ggufv2".
* model_size_in_billions: An integer defining the size of the model in billions of parameters.
* quantizations: A list of strings defining the available quantizations for the model. For PyTorch models, it could be "4-bit", "8-bit", or "none". For ggufv2 models, the quantizations should correspond to values that work with the ``model_file_name_template``.
  Some engines also support ``fp4`` / ``fp8`` / ``bnb`` formats (see :ref:`installation` for backend support details).
   * model_id: A string representing the model ID, possibly referring to an identifier used by Hugging Face. **If model_uri is missing, Xinference will try to download the model from the huggingface repository specified here.**.
   * model_hub: A string representing where to download the model from, like "Huggingface" or "modelscope"
   * model_uri: A string representing the URI where the model can be loaded from, such as "file:///path/to/llama-2-7b". **When the model format is ggufv2, model_uri must be the specific file path. When the model format is pytorch, model_uri must be the path to the directory containing the model files.** If model URI is absent, Xinference will try to download the model from Hugging Face with the model ID.
   * model_revision: A string representing the specific version or commit hash of the model files to use from the repository.
* chat_template: If ``model_ability`` includes ``chat`` , you must configure this option to generate the correct full prompt during chat. This is a Jinja template string. Usually, you can find it in the ``tokenizer_config.json`` file within the model directory.
* stop_token_ids: If ``model_ability`` includes ``chat`` , you can configure this option to control when the model stops during chat. This is a list of integers, and you can typically extract the corresponding values from the ``generation_config.json`` or ``tokenizer_config.json`` file in the model directory.
* stop: If ``model_ability`` includes ``chat`` , you can configure this option to control when the model stops during chat. This is a list of strings, and you can typically extract the corresponding values from the ``generation_config.json`` or ``tokenizer_config.json`` file in the model directory.
* reasoning_start_tag: A special token or prompt used to explicitly instruct the LLM to begin its chain-of-thought or reasoning process in its output.
* reasoning_end_tag: A special token or prompt used to explicitly mark the end of the model's chain-of-thought or reasoning process in its output.
* cache_config: A string representing the parameters and rules for how the system stores and manages temporary data (cache).
* virtualenv: A settings object for model dependency isolation. Please refer to :ref:`this document <virtualenv>` for details.

.. note::
   Besides ``file://``, ``model_uri`` accepts ``oci://<registry>/<repository>:<tag>`` for a model published as a `CNCF ModelPack <https://github.com/modelpack/model-spec>`_ artifact in a container registry. Pulling one requires a running `llmman <https://github.com/llmmanorg/llmman>`_ daemon (``llmman serve``) and the ``llmman`` binary on ``PATH``. Set ``LLMMAN_HOST`` to reach a daemon elsewhere, or ``XINFERENCE_LLMMAN_BIN`` to point at a binary outside ``PATH``.

Register a Custom Model
~~~~~~~~~~~~~~~~~~~~~~~

Register a custom model programmatically:

.. code-block:: python

   import json
   from xinference.client import Client

   with open('model.json') as fd:
       model = fd.read()

   # replace with real xinference endpoint
   endpoint = 'http://localhost:9997'
   client = Client(endpoint)
   client.register_model(model_type="<model_type>", model=model, persist=False)

Or via CLI:

.. code-block:: bash

   xinference register --model-type <model_type> --file model.json --persist

Note that replace the ``<model_type>`` above with ``LLM``, ``embedding`` or ``rerank``. The same as below.


List the Built-in and Custom Models
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

List built-in and custom models programmatically:

.. code-block:: python

   registrations = client.list_model_registrations(model_type="<model_type>")

Or via CLI:

.. code-block:: bash

   xinference registrations --model-type <model_type>

Launch the Custom Model
~~~~~~~~~~~~~~~~~~~~~~~

Launch the custom model programmatically:

.. code-block:: python

   uid = client.launch_model(model_name='custom-llama-2', model_format='pytorch')

Or via CLI:

.. code-block:: bash

   xinference launch --model-name custom-llama-2 --model-format pytorch

Interact with the Custom Model
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Invoke the model programmatically:

.. code-block:: python

   model = client.get_model(model_uid=uid)
   model.generate('What is the largest animal in the world?')

Result:

.. code-block:: json

   {
      "id":"cmpl-a4a9d9fc-7703-4a44-82af-fce9e3c0e52a",
      "object":"text_completion",
      "created":1692024624,
      "model":"43e1f69a-3ab0-11ee-8f69-fa163e74fa2d",
      "choices":[
         {
            "text":"\nWhat does an octopus look like?\nHow many human hours has an octopus been watching you for?",
            "index":0,
            "logprobs":"None",
            "finish_reason":"stop"
         }
      ],
      "usage":{
         "prompt_tokens":10,
         "completion_tokens":23,
         "total_tokens":33
      }
   }

Or via CLI, replace ``${UID}`` with real model UID:

.. code-block:: bash

   xinference generate --model-uid ${UID}

Unregister the Custom Model
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Unregister the custom model programmatically:

.. code-block:: python

   model = client.unregister_model(model_type="<model_type>", model_name='custom-llama-2')

Or via CLI:

.. code-block:: bash

   xinference unregister --model-type <model_type> --model-name custom-llama-2

---

### Doc Source Models Index (`doc/source/models/index.rst`)

.. _models_index:

======
Models
======

List Models
============================

You can list all models of a certain type that are available to launch in Xinference:

.. tabs::

  .. code-tab:: bash shell

    xinference registrations --model-type <MODEL_TYPE> \
                             [--endpoint "http://<XINFERENCE_HOST>:<XINFERENCE_PORT>"] \

  .. code-tab:: bash cURL

    curl http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/model_registrations/<MODEL_TYPE>

  .. code-tab:: python

    from xinference.client import Client
    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
    print(client.list_model_registrations(model_type='<MODEL_TYPE>'))

The following ``MODEL_TYPE`` is supported by Xinference:

.. grid:: 2

    .. grid-item-card::  LLM
      :link: models_llm_index
      :link-type: ref

      Text generation models or large language models

    .. grid-item-card::  embedding
      :link: models_embedding_index
      :link-type: ref

      Text embeddings models

.. grid:: 2

    .. grid-item-card::  image
      :link: models_image_index
      :link-type: ref

      Image generation or manipulation models

    .. grid-item-card::  audio
      :link: models_audio_index
      :link-type: ref

      Audio models

.. grid:: 2

    .. grid-item-card::  rerank
      :link: models_rerank_index
      :link-type: ref

      Rerank models

    .. grid-item-card::  video
      :link: models_video_index
      :link-type: ref

      Video models

.. grid:: 2

    .. grid-item-card::  world
      :link: models_world_index
      :link-type: ref

      World generation models

    .. grid-item-card::  flexible
      :link: flexible
      :link-type: ref

      Flexible models (Traditional ML Models)




You can see all the built-in models supported by xinference :ref:`here <models_builtin_index>`. If the model 
you need is not available, Xinference also allows you to register your own :ref:`custom models <models_custom>`.


Launch and Terminate Model
============================

Each running model instance will be assigned a unique model uid. By default, the model uid is equal to the model name.
This unique id can be used as a handle for the further usage. You can manually assign it by passing ``--model-uid`` option
in the launch command. 

You can launch a model in Xinference either via command line or Xinference's Python client:

.. tabs::

  .. code-tab:: bash shell

    xinference launch --model-name <MODEL_NAME> \
                      [--model-engine <MODEL_ENGINE>] \
                      [--model-type <MODEL_TYPE>] \
                      [--model-uid <MODEL_UID>] \
                      [--endpoint "http://<XINFERENCE_HOST>:<XINFERENCE_PORT>"] \


  .. code-tab:: python

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
    model_uid = client.launch_model(
      model_name="<MODEL_NAME>",
      model_engine="<MODEL_ENGINE>",
      model_type="<MODEL_TYPE>"
      model_uid="<MODEL_UID>"
    )
    print(model_uid)


For model type ``LLM``, launching the model requires not only specifying the model name, but also the size of the parameters
, the model format and the model engine.  Please refer to the list of LLM :ref:`model families <models_llm_index>`.

The following command gives you the currently running models in Xinference:

.. tabs::

  .. code-tab:: bash shell

    xinference list [--endpoint "http://<XINFERENCE_HOST>:<XINFERENCE_PORT>"]


  .. code-tab:: bash cURL

    curl http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/models


  .. code-tab:: python

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
    print(client.list_models())

When you no longer need a model that is currently running, you can remove it in the
following way to free up the resources it occupies:


.. tabs::

  .. code-tab:: bash shell

    xinference terminate --model-uid "<MODEL_UID>" [--endpoint "http://<XINFERENCE_HOST>:<XINFERENCE_PORT>"]

  .. code-tab:: bash cURL

    curl -X DELETE http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/models/<MODEL_UID>


  .. code-tab:: python

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
    client.terminate_model(model_uid="<MODEL_UID>")

.. note::

  For models that are no longer maintained and depend on outdated libraries (such as ``transformers``),
  we recommend enabling the :ref:`Model Virtual Environment <model_virtual_env>` feature
  to ensure they can run properly in a compatible environment.


Model Usage
============================


.. grid:: 2

    .. grid-item-card::  Chat & Generate
      :link: chat
      :link-type: ref

      Learn how to chat with LLMs in Xinference.

    .. grid-item-card::  Tools
      :link: tools
      :link-type: ref

      Learn how to connect LLM with external tools.


.. grid:: 2

    .. grid-item-card::  Embeddings
      :link: embed
      :link-type: ref

      Learn how to create text embeddings in Xinference.

    .. grid-item-card::  Rerank
      :link: rerank
      :link-type: ref

      Learn how to use rerank models in Xinference.


.. grid:: 2

    .. grid-item-card::  Images
      :link: image
      :link-type: ref

      Learn how to generate images with Xinference.

    .. grid-item-card::  Multimodal
      :link: multimodal
      :link-type: ref

      Learn how to process images and audio with LLMs.


.. grid:: 2

    .. grid-item-card::  Audio
      :link: audio
      :link-type: ref

      Learn how to turn audio into text or text into audio with Xinference.

    .. grid-item-card::  Video
      :link: video
      :link-type: ref

      Learn how to generate video with Xinference.

.. grid:: 2

    .. grid-item-card::  World
      :link: world
      :link-type: ref

      Learn how to generate environment videos with world models in Xinference.

    .. grid-item-card::  flexible
      :link: flexible
      :link-type: ref

      Learn how to inference traditional ML models with Xinference.


.. toctree::
   :maxdepth: 2

   xinference_models_hub
   model_abilities/index
   builtin/index
   custom
   model_update
   sources/sources
   virtualenv
   lora
   model_memory

---

### Doc Source Models Lora (`doc/source/models/lora.rst`)

.. _lora:

================
LoRA Integration
================

Currently, Xinference supports launching ``LLM`` and ``image`` models with an attached LoRA fine-tuned model.

Usage
#####

Launch
======
Different from built-in models, xinference currently does not involve managing LoRA models.
Users need to first download the LoRA model themselves and then provide the storage path of the model files to xinference.

.. tabs::

  .. code-tab:: bash shell

    xinference launch <options> 
    --lora-modules <lora_name1> <lora_model_path1>
    --lora-modules <lora_name2> <lora_model_path2>
    --image-lora-load-kwargs <load_params1> <load_value1>
    --image-lora-load-kwargs <load_params2> <load_value2>
    --image-lora-fuse-kwargs <fuse_params1> <fuse_value1>
    --image-lora-fuse-kwargs <fuse_params2> <fuse_value2>

  .. code-tab:: python

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

    lora_model1={'lora_name': <lora_name1>, 'local_path': <lora_model_path1>}
    lora_model2={'lora_name': <lora_name2>, 'local_path': <lora_model_path2>}
    lora_models=[lora_model1, lora_model2]
    image_lora_load_kwargs={'<load_params1>': <load_value1>, '<load_params2>': <load_value2>},
    image_lora_fuse_kwargs={'<fuse_params1>': <fuse_value1>, '<fuse_params2>': <fuse_value2>}

    peft_model_config = {
    "image_lora_load_kwargs": image_lora_load_params,
    "image_lora_fuse_kwargs": image_lora_fuse_params,
    "lora_list": lora_models
    }
    
    client.launch_model(
        <other_options>,
        peft_model_config=peft_model_config
    )


Apply
=====
For LLM models, you can only configure one lora model you want when you use the model.
Specifically, specify that the ``lora_name`` parameter be configured in the ``generate_config``.
``lora_name`` corresponds to the name of the lora in the LAUNCH procedure described above.

.. tabs::

  .. code-tab:: python

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
    model = client.get_model("<model_uid>")
    model.chat(
        messages=[{"role": "user", "content": ""}],
        generate_config={"lora_name": "<your_lora_name>"}
    )


Note
####

* The options ``image_lora_load_kwargs`` and ``image_lora_fuse_kwargs`` are only applicable to models with model_type ``image``.
  They correspond to the parameters in the ``load_lora_weights`` and ``fuse_lora`` interfaces of the ``diffusers`` library.
  If launching an LLM model, these parameters are not required.

* You need to add the parameter lora_name during inference to specify the corresponding lora model. You can specify it in the Additional Inputs option.

* For LLM chat models, currently only LoRA models are supported that do not change the prompt style.

* When using GPU, both LoRA and its base model occupy the same devices.

---

### Doc Source Models Model Memory (`doc/source/models/model_memory.rst`)

.. _cal_model_memory:

========================
Model Memory Calculation
========================

For better planning of VMEM usage, xinference provided tool for model memory calculation: ``cal-model-mem``

Use algorithm from https://github.com/RahulSChand/gpu_poor

Output: model_mem, kv_cache, recurrent_state_mem, prefix_cache_mem, overhead, active_mem

The total includes recurrent state and retained prefix cache. Activation memory
includes vision workspace when present, so vision workspace is not added again.
Sliding attention uses a full-attention cache upper bound. Unsupported layouts
or missing checkpoint-size facts produce an explanatory error instead of an estimate.

Example: 
To calculate memory usage for qwen1.5-chat, run the following command:

.. tabs::

  .. code-tab:: bash Command
    
    xinference cal-model-mem -s 7 -q Int4 -f gptq -c 16384 -n qwen1.5-chat
    
  .. code-tab:: bash Output
    
    model_name: qwen1.5-chat
    kv_cache_dtype: 16
    model size: 7.0 B
    quant: Int4
    context: 16384
    gpu mem usage:
      model mem: 4139 MB
      kv_cache: 8192 MB
      recurrent state: 0 MB
      retained prefix cache: 0 MB
      overhead: 650 MB
      active: 17024 MB
      total: 30005 MB (30 GB)

Syntax
------

* --size-in-billions {model_size}
  
  
  * -s {model_size}


  Set the model size.
  Specify the model size in billions of parameters. Format accept 1_8 and 1.8.
  For example, 7 for 7.0B model size.


* --quantization {precision}
  
  
  * -q {precision} *(Optional)*


  Define the quantization settings for the model.
  For example, Int4 for INT4 quantization.


* --model-name {model_name}
  
  
  * -n {model_name} *(Optional)*


  Specify the model's name.
  If provided, fetch model config from huggingface/modelscope; If not specified, use default model layer to estimate.

 
* --context-length {context_length}
  
  
  * -c {context_length}


  Specify the maximum number of tokens(context length) that your model support.

 
* --model-format {format}
  
  
  * -f {format}


  Specify the format of the model, e.g. pytorch, ggmlv3, etc.

 
.. note::
  The environment variable ``HF_ENDPOINT`` could set the endpoint of HuggingFace. e.g. hf-mirror, etc.
  Please refer to :ref:`this document <models_download>`

---

### Doc Source Models Model Update (`doc/source/models/model_update.rst`)

.. _model_update:

============
Model Update
============
.. versionadded:: v1.13.0

This section briefly introduces a common operation on the "Launch Model" page: updating the model list. It corresponds to the "Type Selection + Update" button at the top of the page, which is used to quickly refresh models of a specific type.

Model update rely on the online model list service provided by :ref:`xinference_models_hub` .

.. raw:: html

    

Update Models
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Operation Location: "Type Selection" dropdown and "Update" button at the top right of the page.
- Usage:
1. Select a model type from the dropdown (such as llm, embedding, rerank, image, audio, video).
2. Click the "Update" button, the page will send an update request to the backend, then automatically jump to the corresponding Tab and refresh the model list of that type.

---

### Doc Source Models Virtualenv (`doc/source/models/virtualenv.rst`)

.. _model_virtual_env:
.. _virtualenv:

==========================
Model Virtual Environments
==========================

.. versionadded:: v1.5.0

Background
##########

Some models are no longer maintained after their release, and the versions of the libraries they depend on remain outdated.
For example, the ``GOT-OCR2`` model still relies on ``transformers`` version 4.37.2. If this library is updated to a newer version,
the model can no longer function properly. On the other hand, many newer models require the latest version of ``transformers``.
This version mismatch leads to dependency conflicts.

Solution
########

To address this issue, we have introduced the **Model Virtual Environment** feature.

Model virtual environments are included in the base Xinference installation:

.. code-block:: bash

    pip install xinference

The feature is enabled by default. You do not need to install engine extras
before launching a model.

Example usage:

.. code-block:: bash

  # For command line
  XINFERENCE_ENABLE_VIRTUAL_ENV=1 xinference-local ...

  # For Docker
  docker run -e XINFERENCE_ENABLE_VIRTUAL_ENV=1 ...

.. warning::

  This feature requires internet access or a self-hosted PyPI mirror.

  Xinference will by default inherit the config for current pip.

.. note::

  Note: When launching a vLLM/SgLang engine model inside a virtual environment, if you encounter
  a cuDNN error, you can set:

  .. code-block:: bash

    export LD_LIBRARY_PATH=$CONDA_PREFIX/lib/python3.12/site-packages/nvidia/cudnn/lib:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=$CONDA_PREFIX/lib/python3.12/site-packages/nvidia/cusparselt/lib:$LD_LIBRARY_PATH
    export LD_LIBRARY_PATH=$CONDA_PREFIX/lib/python3.12/site-packages/nvidia/nccl/lib:$LD_LIBRARY_PATH

.. versionchanged:: v2.0.0

  Starting from **Xinference v2.0**, the model virtual environment feature is
  enabled by default (i.e., ``XINFERENCE_ENABLE_VIRTUAL_ENV`` defaults to ``1``).

  To disable it globally, set ``XINFERENCE_ENABLE_VIRTUAL_ENV=0`` when starting Xinference.

When enabled, Xinference will automatically create a dedicated virtual environment for each model when it is loaded,
and install its specific dependencies there. This prevents dependency conflicts between models,
allowing them to run in isolation without affecting one another.

Using Virtual Environments (v2.0)
#################################

Global toggle
~~~~~~~~~~~~~

Virtual environments are enabled by default starting from v2.0. You can still override this globally:

.. code-block:: bash

  # Enable globally (default)
  XINFERENCE_ENABLE_VIRTUAL_ENV=1 xinference-local -H 0.0.0.0 -p 9997

  # Disable globally
  XINFERENCE_ENABLE_VIRTUAL_ENV=0 xinference-local -H 0.0.0.0 -p 9997

Per-model override at launch time
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can override the global setting when launching a model:

.. code-block:: bash

  # Force enable for this model
  xinference launch -n qwen2.5-instruct --model-engine transformers --enable-virtual-env

  # Force disable for this model
  xinference launch -n qwen2.5-instruct --model-engine transformers --disable-virtual-env

Add or override packages at launch time
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Use ``--virtual-env-package`` (or ``-vp``) multiple times:

.. code-block:: bash

  xinference launch -n qwen2.5-instruct --model-engine transformers \
    --virtual-env-package transformers==4.46.3 \
    --virtual-env-package accelerate==0.33.0

If you specify a package that already exists in the model's default virtualenv package list,
your version replaces the default instead of being appended.


Use local wheel directories at launch time
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Administrators can expose worker-local wheel directories for per-launch package
installation. By default, request-level paths must be located under
``${XINFERENCE_HOME}/wheels``. Configure one or more allowed roots on every
worker with the platform path separator (``:`` on Linux/macOS and ``;`` on
Windows):

.. code-block:: bash

  export XINFERENCE_VIRTUAL_ENV_FIND_LINKS_ALLOWED_ROOTS=/srv/xinference/wheels:/opt/model-wheels

Then pass an absolute directory path when launching the model:

.. code-block:: bash

  xinference launch -n qwen2.5-instruct --model-engine transformers \
    --virtual-env-package custom-kernel==1.0.0 \
    --virtual-env-find-link /srv/xinference/wheels

The REST and Python clients use the ``virtual_env_find_links`` field:

.. code-block:: python

  client.launch_model(
      model_name="qwen2.5-instruct",
      model_engine="transformers",
      virtual_env_packages=["custom-kernel==1.0.0"],
      virtual_env_find_links=["/srv/xinference/wheels"],
  )

Paths are validated by the selected worker after resolving symbolic links. They
must exist, be readable directories, and remain inside an administrator-configured
allowed root. URLs and relative paths are rejected. In a distributed deployment,
the path must be available on every worker that may launch a replica or shard.
Set ``XINFERENCE_VIRTUAL_ENV_FIND_LINKS_ALLOWED_ROOTS`` to an empty value to
disable request-level local wheel directories. Allowed roots should not be
writable by untrusted users because Python wheels execute code during installation
and import.


Storage Location
################

By default, the model’s virtual environment is stored under path:

* Before v1.6.0: :ref:`XINFERENCE_HOME <environments_xinference_home>` / virtualenv / {model_name}
* From v1.6.0 to v1.13.0: :ref:`XINFERENCE_HOME <environments_xinference_home>` / virtualenv / v2 / {model_name}
* Since v1.14.0: :ref:`XINFERENCE_HOME <environments_xinference_home>` / virtualenv / v3 / {model_name} / {python_version}
* Since v2.0: :ref:`XINFERENCE_HOME <environments_xinference_home>` / virtualenv / v4 / {model_name} / {model_engine} / {python_version}

Skip Installed Libraries
########################

.. _skip_installed_libraries:

.. versionadded:: v1.8.1

   This feature requires ``xoscar >= 0.7.12``, which is the minimum Xoscar version required for Xinference v1.8.1.

``xinference`` uses the ``uv`` tool to create virtual environments, with the current Python **system site-packages** set as the base environment.
By default, ``uv`` **does not check for existing packages in the system environment** and reinstalls all dependencies in the virtual environment.
This ensures better isolation from system packages but can result in redundant installations, longer setup times, and increased disk usage.

Starting from ``v1.8.1``, an **experimental feature** is available:
by setting the environment variable ``XINFERENCE_VIRTUAL_ENV_SKIP_INSTALLED=1``, ``uv`` will **skip packages already available in system site-packages**.

.. versionchanged:: v2.0

    This feature is enabled by default in ``v2.0``. To disable it, set ``XINFERENCE_VIRTUAL_ENV_SKIP_INSTALLED=0``.

Advantages
~~~~~~~~~~

- Avoid redundant installations of large dependencies (e.g., ``torch`` + ``CUDA``).
- Speed up virtual environment creation.
- Reduce disk usage.

Usage
~~~~~

.. code-block:: bash

   # Enable experimental feature

   # For command line
   XINFERENCE_ENABLE_VIRTUAL_ENV=1 XINFERENCE_VIRTUAL_ENV_SKIP_INSTALLED=1 xinference-local ...
   # For docker
   docker run -e XINFERENCE_ENABLE_VIRTUAL_ENV=1 -e XINFERENCE_VIRTUAL_ENV_SKIP_INSTALLED=1 ...

Performance Comparison
~~~~~~~~~~~~~~~~~~~~~~

Using the ``CosyVoice 0.5B`` model as an example:

**Without this feature enabled**::

    Installed 98 packages in 187ms
     + aiohappyeyeballs==2.6.1
     + aiohttp==3.12.13
     ...
     + torch==2.7.1
     ...
     + yarl==1.20.1
     + zipp==3.23.0

**With this feature enabled**::

    Installed 7 packages in 12ms
     + diffusers==0.29.0
     + hf-xet==1.1.5
     + huggingface-hub==0.33.2
     + importlib-metadata==8.7.0
     + pillow==11.3.0
     + typing-extensions==4.14.0
     + urllib3==2.5.0


.. _model_launching_virtualenv:

Model Launching: Toggle Virtual Environments and Customize Dependencies
#######################################################################

.. versionadded:: v1.8.1

Starting from v1.8.1, we support toggling the virtual environment for individual model launching,
as well as overriding the model's default settings with custom package dependencies.

Toggle Virtual Environment
~~~~~~~~~~~~~~~~~~~~~~~~~~

When loading a model, you can specify whether to enable the model's virtual environment.
If not specified, the setting will follow the environment variable configuration.

For the Web UI, this can be toggled on or off through the optional settings switch.

.. raw:: html

    

For command-line loading, use the ``--enable-virtual-env`` option to enable the virtual environment, or ``--disable-virtual-env`` to disable it.

Example usage:

.. code-block:: bash

  xinference launch xxx --enable-virtual-env

Set Virtual Environment Package Dependencies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For supported models, Xinference has already defined the package dependencies and version requirements within the virtual environment.
However, if you need to specify particular versions or install additional dependencies, you can manually provide them during model loading.

In the Web UI, you can add custom dependencies by clicking the plus icon in the same location as the virtual environment toggle.

For the command line, use ``--virtual-env-package`` or ``-vp`` to specify a single package version.

Example usage:

.. code-block:: bash

  xinference launch xxx --virtual-env-package transformers==4.54.0

In addition to the standard way of specifying package dependencies, such as ``transformers==xxx``, Xinference also supports some extended syntax.

* ``#system_xxx#``: Using the same version as the system site packages, such as ``#system_numpy#``,
  ensures that the installed package matches the system site package version of numpy. This helps prevent dependency conflicts.


.. _manage_virtual_enviroments:

Manage Virtual Enviroments
##########################

.. versionadded:: v1.14.0

Xinference provides comprehensive virtual environment management for model dependencies,
allowing you to create isolated Python environments for each model with specific package requirements.

.. raw:: html

    

.. raw:: html

    

Key Features
############

**Multiple Python Version Support**:
Each model can have virtual environments
with different Python versions (e.g., Python 3.10.18, 3.11.5),
enabling compatibility with various model requirements.

**Isolated Dependencies**:
Each virtual environment contains its own set of packages,
preventing conflicts between different models' requirements.

Management Operations
#####################

**Listing Virtual Environments**:
View all virtual environments across your cluster,
filtered by model name or worker IP address.

**Creating Environments**:
Automatically created when launching models with enable_virtual_env=true.
The system detects your current Python version and creates an isolated
environment with the required packages.

**Removing Environments**:
Delete specific virtual environments by model name and optionally
Python version, or remove all environments for a model.

ModelHub JSON for Xinference Models
###################################

If you plan to add a model to a model hub for Xinference, define a ``virtualenv`` block
in the model JSON. Starting from v2.0 (v4 flow), **engine-aware markers are recommended**
so one JSON can cover multiple engines.

Important rule:
If a new model supports a specific engine, you **must** include at least one package
entry for that engine in ``virtualenv.packages`` and attach a marker, for example
``#engine# == "vllm"``. Engine availability checks rely on these markers when
virtual environments are enabled.

Minimal virtualenv block (recommended)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code-block:: json

  {
    "virtualenv": {
      "packages": [
        "#transformers_dependencies# ; #engine# == \"transformers\"",
        "#vllm_dependencies# ; #engine# == \"vllm\"",
        "#sglang_dependencies# ; #engine# == \"sglang\"",
        "#llama_cpp_dependencies# ; #engine# == \"llama.cpp\"",
        "#mlx_dependencies# ; #engine# == \"mlx\"",
        "#system_numpy# ; #engine# == \"vllm\""
      ]
    }
  }

Field reference
^^^^^^^^^^^^^^^

- ``packages`` (required): list of pip requirement strings or markers.
- ``inherit_pip_config`` (default ``true``): inherit system pip configuration if present.
- ``index_url`` / ``extra_index_url`` / ``find_links`` / ``trusted_host``:
  pip index and mirror controls.
- ``index_strategy``: passed through to the virtualenv installer (used by some engines).
- ``no_build_isolation``: pip build isolation switch for tricky builds.

Engine placeholders
^^^^^^^^^^^^^^^^^^

Use wrapped placeholders to inject engine defaults:

- ``#vllm_dependencies#``
- ``#sglang_dependencies#``
- ``#mlx_dependencies#``
- ``#transformers_dependencies#``
- ``#llama_cpp_dependencies#``
- ``#diffusers_dependencies#``
- ``#sentence_transformers_dependencies#``

Markers and case
^^^^^^^^^^^^^^^^

Markers use ``#engine#`` or ``#model_engine#`` comparisons (case-sensitive).
Engine values are passed in lowercase internally, so prefer lowercase values,
for example ``#engine# == "vllm"`` or ``#engine# == "transformers"``.

---

### Doc Source Models Xinference Models Hub (`doc/source/models/xinference_models_hub.rst`)

.. _xinference_models_hub:

=====================
Xinference Models Hub
=====================
.. versionadded:: v1.13.0

Overview
########

The `Xinference Models Hub <https://model.xinference.io>`_ is Xinference’s unified platform for model management and collaboration. It provides end-to-end support for model browsing, registration, review, updates, and collaborative maintenance, serving both regular users and model maintainers.

Before the introduction of the Models Hub, model JSON files were manually submitted and modified directly through PRs in the open-source repository (`xorbitsai/inference`). This approach resulted in uncontrolled versioning, long iteration cycles, and delays in delivering updated models—since model updates were tied to product release cycles, users could not obtain new models promptly.

With centralized online model management, the Xinference Models Hub requires that **all model information—including metadata, parameters, and the README—be edited within the platform**. Based on modifications made by :ref:`model maintainers <model_maintainer_guide>`, the system **automatically generates and submits PRs** to the `xorbitsai/inference` repository, ensuring a standardized, automated, and traceable workflow that eliminates inconsistencies caused by manual edits.

Users can obtain the latest model list at any time through the :ref:`model_update` feature, significantly improving model delivery efficiency and overall experience.

User Guide for Regular Users
############################

This section introduces the basic features available to regular registered users.

**Audience:** Regular users without model registration or maintenance permissions.

Core Features
^^^^^^^^^^^^^

Regular users can browse public models without logging in:

* **Access:** `Xinference Models Hub <https://model.xinference.io>`_

Browse Models
~~~~~~~~~~~~~

* **Function:** View all publicly available models
* **Location:** Navigation bar → “Models”
* **Model Details:** The default tab is “README,” which includes model description, usage guide, and important notes

.. note::
   Certain advanced or enterprise-level models are visible only to authorized users.

.. _model_maintainer_guide:

Guide for Model Maintainers
############################

This section describes the features available to users with model registration or maintenance permissions, including model registration, updates, and review workflows.

**Audience:** Users with model registration or maintenance permissions.

Core Features (Login Required)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Model maintainers have access to the following advanced features in addition to the capabilities available to regular users.

User Center
~~~~~~~~~~~

* **Function:** View and manage personal information
* **Location:** Top-right avatar → “User Center”
* **Account Management:** Update profile, email, and other information
* **Token Management:** Configure a personal GitHub Token for model submissions or updates

.. note::
   If no GitHub Token is configured, the system will use a default Token when generating PRs.

Model Registration
~~~~~~~~~~~~~~~~~~

* **Function:** Register new models and submit them for review
* **Location:** After logging in → Top-right avatar → “Model Registration”

**Submission Steps:**

1. Fill in basic model information (name, engine, format, etc.)
2. Edit the README (click “Get README” to auto-generate a template)
3. Submit the model (enable the “Public Model” parameter if registering a public model)

.. note::
   When registering a public model, the system automatically creates a PR in the `xorbitsai/inference` repository.

My Models
~~~~~~~~~

* **Function:** View all models associated with the current account
* **Location:** After logging in → Top-right avatar → “My Models”

Model Maintenance
~~~~~~~~~~~~~~~~~

* **Function:** Modify the model’s JSON or README
* **Location:** Model details page → “Settings” icon

.. note::
   When updating a public model, the system automatically creates a PR in the `xorbitsai/inference` repository.

Review Workflow
~~~~~~~~~~~~~~~

**Submitter Workflow:**

1. Submit a model
2. Wait for review
3. Revise based on reviewer feedback
4. Updated PRs are generated automatically (for public models)

**Reviewer Permissions:** Access to the model review list and model review privileges.

**Reviewer Workflow:**

1. Enter the “Review List”
2. Evaluate the model’s quality, completeness, and compliance
3. Approve or reject, providing feedback as necessary

---

### Doc Source Reference Index (`doc/source/reference/index.rst`)

.. _reference_index:

=============
API Reference
=============


Client
~~~~~~~
.. autosummary::
   :toctree: generated/

   xinference.client.Client

   xinference.client.Client.describe_model
   xinference.client.Client.get_model
   xinference.client.Client.get_model_registration
   xinference.client.Client.get_launch_model_progress
   xinference.client.Client.cancel_launch_model
   xinference.client.Client.get_instance_info
   xinference.client.Client.launch_model
   xinference.client.Client.list_model_registrations
   xinference.client.Client.list_models
   xinference.client.Client.list_cached_models
   xinference.client.Client.list_deletable_models
   xinference.client.Client.confirm_and_remove_model
   xinference.client.Client.query_engine_by_model_name
   xinference.client.Client.register_model
   xinference.client.Client.terminate_model
   xinference.client.Client.abort_request
   xinference.client.Client.vllm_models
   xinference.client.Client.login
   xinference.client.Client.get_workers_info
   xinference.client.Client.get_supervisor_info
   xinference.client.Client.get_progress
   xinference.client.Client.abort_cluster
   xinference.client.Client.unregister_model


Model Handles
~~~~~~~~~~~~~


ChatModelHandle
^^^^^^^^^^^^^^^
.. autosummary::
   :toctree: generated/

   xinference.client.handlers.ChatModelHandle

   xinference.client.handlers.ChatModelHandle.chat
   xinference.client.handlers.ChatModelHandle.generate


EmbeddingModelHandle
^^^^^^^^^^^^^^^^^^^^
.. autosummary::
   :toctree: generated/

   xinference.client.handlers.EmbeddingModelHandle

   xinference.client.handlers.EmbeddingModelHandle.create_embedding


RerankModelHandle
^^^^^^^^^^^^^^^^^
.. autosummary::
   :toctree: generated/

   xinference.client.restful.restful_client.RESTfulRerankModelHandle

   xinference.client.restful.restful_client.RESTfulRerankModelHandle.rerank


GenerateModelHandle
^^^^^^^^^^^^^^^^^^^
.. autosummary::
   :toctree: generated/

   xinference.client.handlers.GenerateModelHandle

   xinference.client.handlers.GenerateModelHandle.generate


ImageModelHandle
^^^^^^^^^^^^^^^^
.. autosummary::
   :toctree: generated/

   xinference.client.handlers.ImageModelHandle

   xinference.client.handlers.ImageModelHandle.text_to_image


AudioModelHandle
^^^^^^^^^^^^^^^^
.. autosummary::
   :toctree: generated/

   xinference.client.handlers.AudioModelHandle

   xinference.client.handlers.AudioModelHandle.transcriptions
   xinference.client.handlers.AudioModelHandle.translations
   xinference.client.handlers.AudioModelHandle.speech


FlexibleModelHandle
^^^^^^^^^^^^^^^^^^^
.. autosummary::
   :toctree: generated/

   xinference.client.restful.restful_client.RESTfulFlexibleModelHandle

   xinference.client.restful.restful_client.RESTfulFlexibleModelHandle.infer


VideoModelHandle
^^^^^^^^^^^^^^^^
.. autosummary::
   :toctree: generated/

   xinference.client.handlers.VideoModelHandle

   xinference.client.handlers.VideoModelHandle.text_to_video


WorldModelHandle
^^^^^^^^^^^^^^^^
.. autosummary::
   :toctree: generated/

   xinference.client.handlers.WorldModelHandle

   xinference.client.handlers.WorldModelHandle.generate

---

### Doc Source User Guide Anthropic (`doc/source/user_guide/anthropic.rst`)

.. _anthropic_api:

Anthropic API compatibility
===========================

Xinference exposes Anthropic-compatible Messages APIs for already launched
models. Use the model UID in requests. The available features depend on the
model and engine.

Endpoints and authentication
----------------------------

Send message requests to ``/v1/messages`` and token-count requests to
``/v1/messages/count_tokens``. The standard routes require
``anthropic-version: 2023-06-01``. The ``/anthropic/v1/messages`` and
``/anthropic/v1/messages/count_tokens`` aliases remain available.

Use your Xinference API key when authentication is enabled. An arbitrary
non-empty key is sufficient when authentication is disabled.

Claude Code
-----------

Launch a model with chat and tool-calling support first (see :ref:`launch`),
then configure Claude Code to use the Xinference server and model UID:

.. code-block:: bash

    export ANTHROPIC_BASE_URL="http://localhost:9997"
    export ANTHROPIC_API_KEY="YOUR_XINFERENCE_API_KEY"
    export ANTHROPIC_MODEL="YOUR_LAUNCHED_MODEL_UID"
    export ANTHROPIC_DEFAULT_HAIKU_MODEL="$ANTHROPIC_MODEL"
    export ANTHROPIC_DEFAULT_SONNET_MODEL="$ANTHROPIC_MODEL"
    export ANTHROPIC_DEFAULT_OPUS_MODEL="$ANTHROPIC_MODEL"
    claude --model "$ANTHROPIC_MODEL"

Python SDK
----------

.. code-block:: python

    import anthropic

    model_uid = "YOUR_LAUNCHED_MODEL_UID"
    client = anthropic.Anthropic(
        api_key="YOUR_XINFERENCE_API_KEY",
        base_url="http://localhost:9997",
    )
    message = client.messages.create(
        model=model_uid,
        max_tokens=1024,
        messages=[{"role": "user", "content": "Hello, Claude"}],
    )
    print(message.content)

Token counting
--------------

The standard ``/v1/messages/count_tokens`` endpoint and its
``/anthropic/v1/messages/count_tokens`` alias use the loaded model's chat
template and tokenizer, including tool definitions and images where supported.
Both routes use the same model permissions and Xinference API key as message
generation.

SGLang P/D decode replicas support text token counting; image token counting
returns HTTP 501.

.. code-block:: python

    token_count = client.messages.count_tokens(
        model=model_uid,
        messages=[{"role": "user", "content": "Hello, Claude"}],
    )
    print(token_count.input_tokens)

Images and documents
--------------------

Messages support text, base64 or URL images, tool results containing images,
and tool references to tools declared in the request. Plain-text documents are
supported; PDF pages are rendered as images with ``pypdfium2`` (install the
``image`` extra), require a vision model, and are limited to 20 pages per
request. File IDs and document citations are not supported.

The request-wide safety limit is 64 rendered images, counting direct images,
tool-result images and PDF pages together. A lower deployment limit still
applies. Exceeding either limit returns HTTP 400 before generation, including
for streaming requests.

Xinference's vLLM vision models default to two images per prompt. The limit
includes images in conversation history and tool results, so successive
single-image messages can also exceed it. Set ``limit_mm_per_prompt`` when
launching the model, for example ``{"image": 8, "video": 0}``, to allow more
images within the model's context length and available GPU memory.

PDF limits apply to the entire request: at most 20 pages and 20 million
rendered pixels. Decoded PDF bytes and rendered PNG bytes each use
``XINFERENCE_MEDIA_MAX_BYTES`` (64 MiB by default). Each PDF block is rendered
once per request, and model access is checked before fetching or rendering content.

Thinking
--------

Adaptive thinking enables the model's reasoning mode. The ``effort`` option
maps to prompt guidance, with ``medium`` preserving the default prompt.
Explicit thinking responses carry Xinference signatures, which can be replayed
with the same model UID and signing key; configure
``XINFERENCE_ANTHROPIC_THINKING_SECRET`` consistently across API servers.
SGLang enforces manual ``budget_tokens`` with its request-level
``max_thinking_tokens`` control when strict thinking is enabled. Unsupported
versions or disabled strict thinking return HTTP 501 for a manual budget.
vLLM uses native ``thinking_token_budget`` with boundaries from Xinference's
reasoning parser; unsupported versions or conflicting boundaries return HTTP 501.
MLX enforces the budget through request-local token constraints. All three engines
close thinking when the budget is reached and continue generating the answer within
``max_tokens``.

Stale, unsigned or foreign thinking blocks are ignored during replay;
accompanying answers and tool calls are preserved.

On physical vLLM models, ``stop_sequences`` applies to streaming and non-streaming
requests, including when thinking is disabled. A stop match during thinking
can end the response before a final answer is produced. A matching
caller-provided string returns ``stop_reason: "stop_sequence"`` and the exact
``stop_sequence``. EOS and token-ID stops return ``end_turn`` with a null
``stop_sequence``.

For physical vLLM models, disconnecting from a streaming or non-streaming
Messages request aborts generation and releases request capacity.

Context management
------------------

``context_management`` supports ``clear_thinking_20251015``. Set ``keep`` to
``"all"`` or ``{"type": "all"}`` to retain all thinking (the default), or use
a positive ``thinking_turns`` value to retain recent thinking turns:

.. code-block:: json

    {
      "context_management": {
        "edits": [{
          "type": "clear_thinking_20251015",
          "keep": {"type": "thinking_turns", "value": 1}
        }]
      }
    }

User messages containing only ``tool_result`` blocks do not start a new
assistant turn. Additional user content starts a new turn. Thinking attached
to unresolved tool calls is retained. Generation, cache-prefix processing
and token counting use the same edited history. Omitting ``edits`` is a no-op;
unsupported edit types return HTTP 400.

Tool choice and structured output
---------------------------------

The vLLM, SGLang and MLX engines support structured JSON,
named and required tool choice through constrained JSON decoding. Disabled
parallel tool use returns at most one tool per response. Forced tool arguments
are buffered until complete.

Xinference automatically connects its existing reasoning parser to
vLLM and SGLang structured decoding for supported reasoning markers and engines with parser
plugin support. Qwen3.5 needs no extra parser configuration. Explicit engine
parser settings are preserved. Manual thinking cannot be combined with forced
tool choice. P/D models return 501 for forced tool choice. Other engines return
501 for unsupported controls. MLX uses ``llguidance`` (included in the
``mlx`` extra) with a fast tokenizer. Requests with constrained output, a manual
thinking budget or explicit cache checkpoints use
request-local decoding; other text requests retain continuous batching.
MLX requires ``mlx-lm>=0.31.3`` for thread-local generation streams.

SGLang counts image tokens through a zero-output native prefill, using the
same image processing as generation. This performs image processing and
prefill work and can warm the engine cache.

With compatible SGLang versions, Xinference enables strict thinking for its
reasoning adapter on the default XGrammar backend, including ordinary requests.
This adds token-filtering overhead, requires a compatible tokenizer and does not
support UNO speculative decoding. Set ``enable_strict_thinking=False`` in model
launch options to disable it; EOS can then end thinking before structured output
begins.

Backend capabilities
--------------------

Xinference checks explicit feature requests against the loaded model and
engine before generation. Unsupported controls return HTTP 501; image input
to a text-only model returns HTTP 400. Translated, emulated, degraded and
rejected compatibility decisions are logged with their reasons. Capability
checks do not add features that the engine cannot implement.

The table summarizes engine support. Supported controls require a
compatible model and engine version, as described above.

.. list-table::
   :header-rows: 1
   :widths: 46 18 18 18

   * - Feature
     - vLLM
     - SGLang
     - MLX
   * - Forced tool choice / disabling parallel tools
     - Supported
     - Supported
     - Supported
   * - Structured JSON
     - Supported
     - Supported
     - Supported
   * - Image token counting
     - Supported
     - Supported
     - Supported
   * - Explicit cache lifetime
     - Automatic caching
     - Automatic caching
     - Text requests (LM and VLM)
   * - Separate reasoning-token budget
     - Supported with matching reasoning boundaries
     - Supported with strict thinking
     - Supported with reasoning markers

Prompt caching
--------------

For text requests, both MLX text and vision models support up to four explicit
``cache_control`` breakpoints. Each checkpoint has its own ``5m`` or ``1h``
lifetime, refreshed on a cache hit. Put longer lifetimes before shorter ones.
Expired checkpoints are removed independently; a live earlier checkpoint can
still be reused after a later one expires. Cache storage obeys the configured
``prompt_cache_size`` and ``prompt_cache_bytes`` capacity limits, so memory
pressure can evict checkpoints before their lifetime ends. Usage reports
actual cache writes and reads.

The limits apply separately to automatic caching and explicit checkpoint
caching; their combined capacity can reach twice the configured limits.
MLX P/D models retain automatic caching.

Image requests, vLLM and SGLang retain automatic engine caching, indicated by
the ``x-xinference-cache-control: automatic`` response header. Their engine
caches do not expose independent per-breakpoint lifetime control.

---

### Doc Source User Guide Audit Security (`doc/source/user_guide/audit_security.rst`)

.. _user_guide_audit_security:

============================
Audit Logging and Security
============================

When the database-backed authentication system is enabled (the default, see
:ref:`user_guide_auth_system`), Xinference records an audit trail of API
activity and protects the API against brute-force attacks. Administrators
can inspect both from the web UI.

Audit logging
=============

Protected and authenticated API activity is recorded as JSON lines in
``<XINFERENCE_LOG_DIR>/audit.log`` (``<XINFERENCE_HOME>/logs`` by default).
Files rotate daily (and by size) and are kept for
``XINFERENCE_AUDIT_LOG_RETENTION_DAYS`` days (default 90).

This is an authentication and security audit trail, not a complete HTTP
access log. Xinference intentionally excludes ``/v1/audit/*``,
``/v1/cluster/auth``, ``/v1/cluster/ui_config``, ``/status``, and
``/v1/address``; public or bootstrap routes that do not pass through the
authenticated audit path may also be absent.

Each entry contains, among others:

* ``category``: ``inference`` (model calls such as ``/v1/chat/...``,
  ``/v1/embeddings``), ``auth`` (login/token endpoints), or ``admin``
  (everything else).
* ``user``, ``auth_type``, ``api_key_name`` / ``api_key_prefix``: who made
  the call and how it was authenticated.
* ``model_id`` / ``model_name`` / ``model_type``: the model involved, if any.
* ``endpoint``, ``status``, ``latency_ms``, ``client_ip``, ``node``,
  ``address``: what was called, the outcome, and where it ran.

Audit Center
------------

The **Audit Center** page of the web UI lets administrators search and
filter the audit trail (by time range, user, API key, model, category,
status, and client IP). It is backed by ``GET /v1/audit/search``, which
requires the ``admin`` permission.

By default the search reads the local ``audit.log``. If the
``XINFERENCE_ES_URL`` environment variable points at an Elasticsearch
cluster (e.g. with the audit log shipped by Filebeat), the search queries
Elasticsearch instead, using the index pattern from
``XINFERENCE_AUDIT_ES_INDEX`` (default ``xinference-audit-*``).
Elasticsearch-backed searches use a point-in-time snapshot and
``search_after`` pagination, so administrators can browse beyond
Elasticsearch's default 10,000-result window without changing
``index.max_result_window``.

Log Center and protected request bodies
---------------------------------------

When ``XINFERENCE_ES_URL`` is configured, the **Log Center** queries the index
pattern selected by ``XINFERENCE_ES_INDEX``. The default remains
``xinference-logs-*`` for compatibility with existing deployments. Operators
that ship ``model_request.log`` into a separate index can set this variable to
a read-only alias, such as ``xinference-log-search``, that spans both the
application-log and model-request indices.

``GET /v1/cluster/logs/correlated`` returns a time-ordered request timeline and
requires ``logs:list``. The query accepts a ``request_id`` and an optional time
range of at most seven days. Metadata queries, including normal log search and
context search, always exclude ``request_body`` and ``request_body_raw`` from
the Elasticsearch response.

Request bodies are fetched separately with
``GET /v1/cluster/model-requests/{request_id}/body``. This endpoint requires
both ``logs:list`` and ``model_requests:read_body`` and is denied when
authentication is disabled. It queries
``XINFERENCE_MODEL_REQUEST_ES_INDEX`` (default
``xinference-model-request-*``), and successful or failed access attempts are
recorded in the admin audit trail. API keys cannot use this administrative
endpoint.

The read-only alias is a routing convenience, not a field-level security
boundary. Browser clients should access logs only through the Xinference API,
and Elasticsearch credentials should not be exposed to users who are not
allowed to read captured request bodies.

Brute-force protection
======================

Failed API-key authentication attempts are rate-limited on two levels:

* **Per IP**: an IP presenting invalid API keys is banned after
  ``XINFERENCE_RATE_LIMIT_IP_MAX_FAILURES`` failures (default 10) within
  ``XINFERENCE_RATE_LIMIT_IP_WINDOW_SECONDS`` (default 300), for
  ``XINFERENCE_RATE_LIMIT_IP_BAN_SECONDS`` (default 3600). Requests from a
  banned IP are rejected with ``429 Too Many Requests``.
* **Per (IP, API key)**: repeated failures with one specific key from one IP
  ban that combination after ``XINFERENCE_RATE_LIMIT_KEY_MAX_FAILURES``
  failures (default 5) within
  ``XINFERENCE_RATE_LIMIT_KEY_WINDOW_SECONDS`` (default 300), for
  ``XINFERENCE_RATE_LIMIT_KEY_BAN_SECONDS`` (default 3600).

Individual API keys can override the key-level limits with the
``rate_limit_max_failures`` / ``rate_limit_window_seconds`` /
``rate_limit_ban_seconds`` fields when creating or updating the key.

Security Settings
-----------------

The **Security Settings** page of the web UI lets administrators:

* view and tune the rate-limit configuration at runtime,
* list currently banned IPs and (IP, key) pairs,
* lift bans individually or all at once.

The equivalent REST endpoints live under ``/v1/admin/security/*`` (e.g.
``GET``/``PUT /v1/admin/security/rate-limit``,
``GET /v1/admin/security/banned-ips``,
``POST /v1/admin/security/unban-ip``) and require the ``admin`` permission.

---

### Doc Source User Guide Auth System (`doc/source/user_guide/auth_system.rst`)

.. _user_guide_auth_system:

=====================================================
Authentication System (database-backed)
=====================================================

.. versionadded:: 3.0
   The database-backed authentication system, enabled by default.

Xinference ships with a database-backed authentication and authorization
system. It stores users, permissions, API keys, and refresh tokens in a
local SQLite database, and supports creating and managing users and API keys
at runtime through REST endpoints — no server restart required.

**Since v3.0, this system is enabled by default** — a fresh Xinference
deployment requires authentication out of the box.

.. note::
   Earlier releases shipped a simpler, in-memory ``--auth-config`` JSON
   system. It has been removed; the database-backed system described here is
   the only authenticated mode.

Enabling / disabling
=====================
Authentication is controlled by the ``XINFERENCE_AUTH_ADVANCED`` environment
variable:

* Unset, or any of ``1`` / ``true`` / ``yes`` (case-insensitive): enabled (the default).
* Any of ``0`` / ``false`` / ``no`` (case-insensitive): disabled — Xinference
  runs with **no authentication at all**, and every endpoint is served
  without a login or API key.

.. code-block:: bash

   # Default: authentication enabled
   xinference-local -H 0.0.0.0

   # Disable authentication entirely
   export XINFERENCE_AUTH_ADVANCED=false
   xinference-local -H 0.0.0.0

Initial admin account
=======================
On first startup with authentication enabled, the user table is empty and
Xinference does **not** create an admin account automatically. Two endpoints
handle first-run setup without requiring an existing login:

* ``GET /v1/admin/setup/status``: returns ``{"needs_setup": true, "initialized": false}``
  while no account exists yet.
* ``POST /v1/admin/setup``: creates the first admin account (with all
  permissions) given a ``username``, ``password``, and the ``setup_token``
  printed in the server startup log.

.. code-block:: bash

    curl -X POST "<endpoint>/v1/admin/setup" \
      -H "Content-Type: application/json" \
      -d '{"setup_token": "<setup-token-from-startup-log>", "username": "admin", "password": "choose-a-strong-password"}'

The web UI redirects to a setup page that asks for the setup token from the
server startup log and the new admin's username and password, then to the
login page.

Only a caller with the setup token can create the initial administrator. The
token is stored in the auth database, shared across API processes using that
database, and remains valid across restarts until setup completes. It is
never returned by the setup status API. Once an account exists,
``/v1/admin/setup`` refuses further calls. An operator with shell access can
recover a lost admin password with ``xinference-reset-auth-password``.

Deleting the final account starts a new setup cycle with a new token in the
server log; consumed tokens stay invalid. Recreating a container without its
auth database also requires the new instance's token.

Resetting a lost admin password
=================================
If the admin password is lost, an operator with shell access to the machine
running the RESTful API can reset an admin password directly against the auth
database, without logging in:

.. code-block:: bash

    xinference-reset-auth-password --username admin

The command prompts for a new password (or accept ``--password`` on the
command line), updates that admin's password, and revokes the user's active
refresh tokens so any stale sessions stop working. It only operates on users
that already hold the ``admin`` permission, and reads the database at
``XINFERENCE_AUTH_DB_PATH`` (override with ``--db-path``). Run
``xinference-reset-auth-password --help`` for all options.

Secrets and storage locations
===============================
The system needs a JWT signing secret and an encryption key (used to encrypt
stored API keys at rest), plus a database file. All three can be overridden
with environment variables; if you don't set them, Xinference generates and
persists them automatically on first run under ``XINFERENCE_HOME``
(``~/.xinference`` by default):

.. list-table::
   :header-rows: 1

   * - Purpose
     - Environment variable
     - Default location
   * - JWT signing secret
     - ``XINFERENCE_AUTH_JWT_SECRET_KEY``
     - ``<XINFERENCE_HOME>/auth/jwt_secret_key`` (auto-generated)
   * - API key encryption key
     - ``XINFERENCE_AUTH_ENCRYPTION_KEY``
     - ``<XINFERENCE_HOME>/auth/encryption_key`` (auto-generated)
   * - User/API key database
     - ``XINFERENCE_AUTH_DB_PATH``
     - ``<XINFERENCE_HOME>/auth/auth.db``

Auto-generated secrets are written once and reused on subsequent restarts,
so existing JWTs and encrypted API keys keep working across restarts. In a
distributed deployment (supervisor + workers), make sure all processes that
run the RESTful API share the same ``XINFERENCE_HOME`` (or set the same
explicit environment variables), so they agree on the same secrets and
database.

Permissions
===========
Xinference defines the following interface permissions:

* ``models:list``: Permission to list models and get models' information.
* ``models:read``: Permission to use models.
* ``models:write``: Permission to launch and stop models.
* ``models:register``: Permission to register and unregister custom models.
* ``keys:create``: Permission to create API keys (for oneself, or for others when combined with ``keys:manage``).
* ``keys:manage``: Permission to list, update, delete, and reveal any user's API keys.
* ``users:manage``: Permission to create, update, delete users, and manage their permissions.
* ``cache:list`` / ``cache:delete``: Permissions to list/delete cached model files.
* ``virtualenv:list`` / ``virtualenv:delete``: Permissions to list/delete per-model virtual environments.
* ``logs:list``: Permission to view cluster logs.
* ``monitor:view``: Permission to view the monitoring dashboards.
* ``settings:read``: Permission to view system settings.
* ``settings:write``: Permission to update and restore system settings.
* ``admin``: Administrators have all of the above.

A caller may only grant permissions they themselves hold — for example, a
user with only ``users:manage`` cannot grant ``admin`` to someone else.

.. note::
   Earlier releases used finer-grained scope names: ``models:start`` and
   ``models:stop`` (now ``models:write``), and ``models:add`` /
   ``models:unregister`` (now ``models:register``). Tokens and API keys
   carrying the legacy names keep working — the server transparently maps
   them to the new scopes — but this compatibility mapping is deprecated and
   will be removed in a future release. Use the new names when granting
   permissions.

Usage
=====
With authentication enabled, all usage remains the same, except for the
addition of a login step at the beginning or using an API key.

Signin
------
Signin for command line users:

.. code-block:: bash

   xinference login -e <endpoint> --username  --password 


For python SDK users:

.. code-block:: python

   from xinference.client import Client
   client = Client('<endpoint>')
   client.login('<name>', '')


Direct HTTP clients must send ``POST /token`` a JSON object with string
``username`` and ``password`` fields. Form-encoded, malformed, or non-object
bodies and non-string credentials return ``400 Bad Request``; incorrect
credentials return ``401 Unauthorized``.

For web UI users, when opening the web UI, you will first be directed to the login page. After logging in, you can use the web UI normally.

Api-Key
-------
For command line users, just add ``--api-key`` or ``-ak`` option in the command you want to use.

.. code-block:: bash

   xinference launch <other options> --api-key <your_api_key>


For python SDK users, pass the ``api_key`` parameter when initializing the client, just like the ``OPENAI`` Python client.

.. code-block:: python

   from xinference.client import Client
   client = Client('<endpoint>', api_key='<your_api_key>')


Xinference is also compatible with the ``OPENAI`` Python SDK as well.

.. code-block:: python

   from openai import OpenAI
   client = OpenAI(base_url="<xinference endpoint>" + "/v1", api_key="<your_api_key>")
   client.models.list()

For http request, pass ``Authorization: Bearer api-key`` in request header.

.. code-block::

    curl --request GET \
      --url "<xinference endpoint>" \
      --header "Authorization: Bearer <your_api_key>"

Managing users and API keys
==============================
Once logged in as a user with the ``users:manage`` and/or ``keys:manage``
permission (the bootstrap ``admin`` account has both), you can manage users
and API keys through REST endpoints under ``/v1/admin``, for example:

.. code-block:: bash

    # Create a new user
    curl -X POST "<endpoint>/v1/admin/users" \
      -H "Authorization: Bearer <admin_access_token>" \
      -H "Content-Type: application/json" \
      -d '{"username": "alice", "password": "s3cret!", "permissions": ["models:list", "models:read"]}'

    # Create an API key for the current user
    curl -X POST "<endpoint>/v1/admin/keys" \
      -H "Authorization: Bearer <access_token>" \
      -H "Content-Type: application/json" \
      -d '{"name": "my-key"}'

API key names are required. Names are trimmed before storage and must be
unique across all users; comparisons ignore ASCII letter case. When an existing
database is upgraded, blank or duplicate names are migrated to deterministic
unique names without changing the API key values.

Other supported endpoints include listing/updating/deleting users
(``/v1/admin/users``, ``/v1/admin/users/{user_id}``), changing a user's
password (``/v1/admin/users/{user_id}/password``), and listing, updating,
deleting, and revealing API keys (``/v1/admin/keys``,
``/v1/admin/keys/{key_id}``, ``/v1/admin/keys/{key_id}/reveal``).

Managing via the web UI
--------------------------
Everything above is also available from dedicated pages in the web UI:

* **User Management**: create, update, disable, and delete users, and edit
  their permissions (requires ``users:manage``).
* **API Key Management**: create, update, delete, and reveal API keys, with
  optional per-model access restrictions (requires ``keys:create`` for your
  own keys, ``keys:manage`` for other users' keys).
* **Security Settings**: view and tune brute-force protection (login/API-key
  failure rate limits) and unban blocked IPs or keys (requires ``admin``).
  See :ref:`user_guide_audit_security` for details.
* **System Settings**: view model download configuration with
  ``settings:read``; saving changes or restoring defaults additionally
  requires ``settings:write``.
* **Audit Center**: browse recorded API activity (requires ``admin``). See
  :ref:`user_guide_audit_security`.

Http Status Code
================
Add the following two HTTP status codes:

* ``401 Unauthorized``: login information or token verifies failed.
* ``403 Forbidden``: No enough permissions when accessing interfaces.

For the command line, SDK, or web UI users, there will be clear information prompts when encountering authorization and permissions issues.

Behavior notes
================

Permission changes take effect without re-login
-------------------------------------------------
Route scope checks read the user's **current** permissions from the
database on every request, not the scopes baked into the JWT at login.
Granting or revoking a permission takes effect on the user's next API
call — no re-login required. This applies to JWT-based browser sessions;
API keys are also live-read (they always have been).

Administrator revocation also takes effect on the next API request.
``GET /v1/auth/me`` returns current account permissions; the Web UI refreshes
them periodically and when the window regains focus.

Local users can change their own password through
``PUT /v1/admin/users/{user_id}/password`` without ``users:manage``.
Supply ``current_password`` and ``new_password``; a forced first password
change only requires ``new_password``. After a password change, sign in again
because existing access and refresh tokens are revoked.

Administrators and callers with ``users:manage`` may reset passwords, including
their own, without ``current_password``. When supplying it for a self-change,
it is verified.

Configurable access-token lifetime
-------------------------------------
The access-token lifetime defaults to 30 minutes and can be overridden
with the ``XINFERENCE_ACCESS_TOKEN_EXPIRE_MINUTES`` environment variable:

.. code-block::

    export XINFERENCE_ACCESS_TOKEN_EXPIRE_MINUTES=10

A shorter lifetime shrinks the token-theft window. The refresh token
lifetime is 7 days and is not currently configurable.

Feedback
==========
This feature is still in an experimental stage.
Feel free to provide feedback on usage issues or improvement suggestions through `GitHub issues <https://github.com/xorbitsai/inference/issues>`_ or
`our Telegram group <https://t.me/+nCNpwmySwk9iYmI1>`_.

---

### Doc Source User Guide Backends (`doc/source/user_guide/backends.rst`)

.. _user_guide_backends:

========
Backends
========

Xinference supports multiple backends for different models. After the user specifies the model,
xinference will automatically select the appropriate backend.

.. raw:: html

   

For speculative decoding methods and launch options, see :ref:`speculative_decoding`.

.. _llama_cpp_backend:

llama.cpp
=========

Xinference now supports `xllamacpp <https://github.com/xorbitsai/xllamacpp>`_ which developed by Xinference team
to run llama.cpp backend.
`llama.cpp` is developed based on the tensor library `ggml`, supporting inference of
the LLaMA series models and their variants.

.. warning::

    Since Xinference v1.5.0,
    ``xllamacpp`` becomes default option for llama.cpp, and ``llama-cpp-python`` is deprecated.
    Since Xinference v1.6.0, ``llama-cpp-python`` has been removed.

Automatic GPU wheel selection
-----------------------------

.. versionadded:: 3.0

When per-model virtual environments are enabled and a usable NVIDIA GPU is
present, Xinference automatically installs the ``xllamacpp`` wheel matching
the detected CUDA line. CUDA 12.8 and later in the 12.x line use the ``cu128``
wheel index, while CUDA 13.x uses the ``cu132`` wheel index. CPU-only hosts and
unsupported CUDA versions keep the default CPU wheel from PyPI. No separate
``xllamacpp`` GPU installation command is required for this launch path.


For all configurable llama.cpp parameters, please refer to the definition of the ``common_params`` structure in ``llama.cpp`` `common.h <https://github.com/ggml-org/llama.cpp/blob/master/common/common.h>`_

There may be some nested parameters. For example, ``sampling.top_k``. Just use the ``.`` to separate nested parameters.

Here is an example of setting nested sampling parameters in WebUI:

.. raw:: html

    

Auto NGL
-------------

.. versionadded:: v1.6.1
    Auto GPU layers estimation is enabled since v1.6.1 when ``n-gpu-layers`` is not specified (default is -1).

This feature automatically detects the number of GPU layers (NGL) for the llama.cpp backend. Please be aware that this
is not an accurate calculation. Therefore, the ``-ngl`` result might not be the most optimized, and there is still a
chance of encountering an out-of-memory error.

Currently, there is no official implementation for auto ngl. Please refer to the following issues for more information:

- https://github.com/ggml-org/llama.cpp/issues/13860
- https://github.com/ggml-org/llama.cpp/pull/6502

Our implementation is based on the Ollama auto ngl, but there are some differences:

- We utilize device information detected by `xllamacpp <https://github.com/xorbitsai/xllamacpp>`_.
- We have removed support for less popular architectures, these architectures will use the default calculation.
- We fall back to offloading all the layers to the GPU if the auto ngl fails.
- We do not support multimodal projectors embedded into the model GGUF, as this is a very experimental feature.


Common Issues
-------------

- **Server error: {'code': 500, 'message': 'failed to process image', 'type': 'server_error'}**

  The error logs from server:

  .. code-block::

    encoding image or slice...
    slot update_slots: id  0 | task 0 | kv cache rm [10, end)
    srv  process_chun: processing image...
    ggml_metal_graph_compute: command buffer 0 failed with status 5
    error: Internal Error (0000000e:Internal Error)
    clip_image_batch_encode: ggml_backend_sched_graph_compute failed with error -1
    failed to encode image
    srv  process_chun: image processed in 2288 ms
    mtmd_helper_eval failed with status 1
    slot update_slots: id  0 | task 0 | failed to process image, res = 1

  This could be caused by running out of memory. You can try reducing memory usage by decreasing ``n_ctx``.

- **Server error: {'code': 400, 'message': 'the request exceeds the available context size. try increasing the context size or enable context shift', 'type': 'invalid_request_error'}**

  If you are using the multimodal feature, the ``ctx_shift`` is disabled by default. Please increase the context size by
  either increasing ``n_ctx`` or reducing ``n_parallel``.

- **Server error: {'code': 500, 'message': 'Input prompt is too big compared to KV size. Please try increasing KV size.', 'type': 'server_error'}**

  The error logs from server:

  .. code-block::

    ggml_metal_graph_compute: command buffer 1 failed with status 5
    error: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
    graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
    llama_decode: failed to decode, ret = -3
    srv  update_slots: failed to decode the batch: KV cache is full - try increasing it via the context size, i = 0, n_batch = 2048, ret = -3

  This could be caused by the KV cache allocation failure. You can try to reduce the context size by either reducing
  ``n_ctx`` or increasing ``n_parallel``, or loading a partial model onto the GPU by adjusting ``n_gpu_layers``. Be aware
  that if you are handling inference requests serially, increasing ``n_parallel`` can't improve the latency or throughput.

transformers
============
Transformers supports the inference of most state-of-art models. It is the default backend for models in PyTorch format.

.. _vllm_backend:

vLLM
====
vLLM is a fast and easy-to-use library for LLM inference and serving.

vLLM is fast with:

- State-of-the-art serving throughput
- Efficient management of attention key and value memory with PagedAttention
- Continuous batching of incoming requests
- Optimized CUDA kernels

When the following conditions are met, Xinference will choose vLLM as the inference engine:

- The model format is ``pytorch``, ``gptq``, ``awq``, ``fp4``, ``fp8`` or ``bnb``.
- When the model format is ``pytorch``, the quantization is ``none``.
- When the model format is ``awq``, the quantization is ``Int4``.
- When the model format is ``gptq``, the quantization is ``Int3``, ``Int4`` or ``Int8``.
- The system is Linux and has at least one CUDA device
- The model family (for custom models) / model name (for builtin models) is within the list of models supported by vLLM

Currently, supported model includes:

.. vllm_start

- ``code-llama``, ``code-llama-instruct``, ``code-llama-python``, ``deepseek``, ``deepseek-chat``, ``deepseek-coder``, ``deepseek-coder-instruct``, ``deepseek-r1-distill-llama``, ``HuatuoGPT-o1-LLaMA-3.1``, ``llama-2``, ``llama-2-chat``, ``llama-3``, ``llama-3-instruct``, ``llama-3.1``, ``llama-3.1-instruct``, ``llama-3.3-instruct``, ``minicpm5-1b``, ``minicpm5-2b``, ``tiny-llama``, ``Yi``, ``Yi-1.5``, ``Yi-1.5-chat``, ``Yi-1.5-chat-16k``, ``Yi-200k``, ``Yi-chat``
- ``codestral-v0.1``, ``mistral-instruct-v0.1``, ``mistral-instruct-v0.2``, ``mistral-instruct-v0.3``, ``mistral-large-instruct``, ``mistral-nemo-instruct``, ``mistral-v0.1``, ``openhermes-2.5``, ``seallm_v2``
- ``Spark-X2.5``, ``Spark-X2.5-Base``
- ``Baichuan-M2``, ``codeqwen1.5``, ``codeqwen1.5-chat``, ``deepseek-r1-distill-qwen``, ``DianJin-R1``, ``fin-r1``, ``HuatuoGPT-o1-Qwen2.5``, ``KAT-V1``, ``marco-o1``, ``qwen1.5-chat``, ``qwen2-instruct``, ``qwen2.5``, ``qwen2.5-coder``, ``qwen2.5-coder-instruct``, ``qwen2.5-instruct``, ``qwen2.5-instruct-1m``, ``qwenLong-l1``, ``QwQ-32B``, ``QwQ-32B-Preview``, ``seallms-v3``, ``skywork-or1``, ``skywork-or1-preview``, ``vibethinker``, ``XiYanSQL-QwenCoder-2504``
- ``llama-3.2-vision``, ``llama-3.2-vision-instruct``
- ``baichuan-2``, ``baichuan-2-chat``
- ``InternLM2ForCausalLM``
- ``qwen-chat``
- ``mixtral-8x22B-instruct-v0.1``, ``mixtral-instruct-v0.1``, ``mixtral-v0.1``
- ``cogagent``
- ``glm-edge-chat``, ``glm4-chat``, ``glm4-chat-1m``
- ``codegeex4``, ``glm-4v``
- ``qwen3.8-max``
- ``seallm_v2.5``
- ``orion-chat``
- ``qwen1.5-moe-chat``, ``qwen2-moe-instruct``
- ``CohereForCausalLM``
- ``deepseek-v2-chat``, ``deepseek-v2-chat-0628``, ``deepseek-v2.5``, ``deepseek-vl2``
- ``deepseek-prover-v2``, ``deepseek-r1``, ``deepseek-r1-0528``, ``deepseek-v3``, ``deepseek-v3-0324``, ``Deepseek-V3.1``, ``moonlight-16b-a3b-instruct``
- ``deepseek-r1-0528-qwen3``, ``qwen3``
- ``minicpm3-4b``
- ``internlm3-instruct``
- ``gemma-3-1b-it``
- ``glm4-0414``
- ``minicpm-2b-dpo-bf16``, ``minicpm-2b-dpo-fp16``, ``minicpm-2b-dpo-fp32``, ``minicpm-2b-sft-bf16``, ``minicpm-2b-sft-fp32``, ``minicpm4``
- ``Ernie4.5``
- ``Qwen3-Coder``, ``Qwen3-Instruct``, ``Qwen3-Thinking``
- ``glm-4.5``, ``GLM-4.6``, ``GLM-4.7``
- ``gpt-oss``
- ``seed-oss``
- ``Qwen3-Next-Instruct``, ``Qwen3-Next-Thinking``
- ``DeepSeek-V3.2``, ``DeepSeek-V3.2-Exp``
- ``MiniMax-M2``, ``MiniMax-M2.5``, ``MiniMax-M2.7``
- ``GLM-4.7-Flash``
- ``glm-5``, ``glm-5.1``, ``glm-5.2``, ``glm-5.3``
- ``DeepSeek-V4-Flash``, ``DeepSeek-V4-Flash-0731``, ``DeepSeek-V4-Pro``
- ``Hy-MT2-1.8B``, ``Hy-MT2-7B``
- ``Hy-MT2-30B-A3B``

.. vllm_end

Besides LLMs, vLLM can also serve embedding models. Model families whose
names start with ``bge``, ``gte``, ``text2vec``, ``m3e``, ``Qwen3``, or
``bce`` can be launched with ``--model-engine vllm`` — see
:ref:`Embeddings `.

vLLM can also serve as the engine for some text-to-image models
(e.g. ``Qwen-Image``, ``Z-Image-Turbo``) on Linux with NVIDIA GPUs, see
:ref:`image engines `. This requires the vLLM-Omni add-on with a vLLM
of the same major.minor version, e.g.
``pip install 'vllm-omni==0.28.*' 'vllm==0.28.*'``.

.. _sglang_backend:

SGLang
======
`SGLang <https://github.com/sgl-project/sglang>`_ has a high-performance inference runtime with RadixAttention.
It significantly accelerates the execution of complex LLM programs by automatic KV cache reuse across multiple calls.
And it also supports other common techniques like continuous batching and tensor parallelism.

Besides LLMs, SGLang can also serve as the engine for some text-to-image models
(e.g. ``Qwen-Image``, ``Z-Image-Turbo``) on Linux with NVIDIA GPUs, see
:ref:`image engines `. This requires SGLang with diffusion support:
``pip install 'sglang[diffusion]>=0.5.20,<0.6'``.

.. _mlx_backend:

MLX
===
`MLX <https://github.com/ml-explore/mlx-examples/tree/main/llms>`_ provides efficient runtime
to run LLM on Apple silicon. It's recommended to use for Mac users when running on Apple silicon
if the model has MLX format support.

MLX can also serve supported audio models on Apple silicon. Whisper models,
``F5-TTS``, and ``Kokoro-82M`` expose it through ``--model-engine MLX``; see
:ref:`Audio engines <audio>`.



.. _trt_llm_backend:

TRT-LLM
========

.. versionadded:: v4.1.0

The TRT-LLM backend uses the Python LLM API inside the model process. It requires Linux, NVIDIA CUDA GPUs and ``tensorrt-llm==1.2.1``. Model support follows the text architectures implemented by that version, including Qwen2/2.5, Qwen3 (dense, MoE and Next), Llama, Mistral/Mixtral, Gemma3, Phi3, DeepSeek, GLM MoE, MiniMax M2, GPT-OSS, Hunyuan, EXAONE, Nemotron, SeedOSS and Starcoder2. Fine-tuned and custom models use the same architecture matching.

Ordinary replicas accept ``pytorch``, ``fp8`` and ``fp4`` checkpoints, with quantization inferred from the checkpoint configuration. This includes compatible ModelOpt exports and native FP8 block-scaled or MXFP4 weights. Use ModelOpt exports for AWQ; generic Hugging Face AWQ/GPTQ, bitsandbytes, GGUF and MLX checkpoint formats are not supported by this PyTorch loader. TRT validates the checkpoint recipe and GPU kernel requirements at startup. Ordinary replicas accept ``dtype="auto"``, ``"float16"`` or ``"bfloat16"``; TP=PP=CP=1 still applies. Shared caching and P/D have the additional constraints below.

The example launches two independent replicas with a shared 512 MiB Xavier CPU cache. Each replica uses one GPU. Install TRT-LLM in a compatible CUDA environment and set ``enable_virtual_env=False``, or use a per-model virtual environment with matching dependencies. Each replica currently requires TP=PP=CP=1, with TRT's worker running directly in the Xinference model process. Qwen3 weights and KV pages use BF16.

.. code-block:: python

   from xinference.client import Client

   client = Client("http://localhost:9997")
   client.launch_model(
       model_uid="qwen-trt",
       model_name="qwen3",
       model_size_in_billions="0_6",
       model_format="pytorch",
       model_engine="TRT-LLM",
       quantization="none",
       replica=2,
       n_gpu=1,
       context_length=4096,
       enable_virtual_env=False,
       enable_xavier=True,
       xavier_cache_bytes=512 * 1024**2,
   )

Xavier publishes complete 64-token BF16 KV pages and restores matching prefixes in another replica through actor RPC. Partial pages and the final prompt token are computed locally. The shared cache uses bounded storage and LRU eviction; transfers currently use synchronous CPU staging.

Shared caching and same-engine P/D require TP=PP=CP=1, BF16 weights and KV, and disabled chunked prefill and overlap scheduling. Quantization, hybrid attention, multimodal input, LoRA, speculative decoding and ``cache_salt`` are not supported in these configurations. Ordinary replicas support :ref:`speculative_decoding` with NGram or Draft/Target. For supported cross-engine P/D combinations, see :ref:`user_guide_pd_separation`.

For same-engine P/D, set explicit prefill/decode roles in ``replica_config`` as described in :ref:`user_guide_pd_separation`. The default ``transfer_backend_type="xavier"`` uses NIXL to transfer KV pages directly between GPUs. Set ``transfer_backend_type="nixl"`` for TRT's native context/generation protocol and NIXL transport; do not also set ``enable_xavier=True``. Both paths support generation, chat, streaming and cancellation through the same Xinference API.

The shared cache actor reports ``imported_pages`` after copies into the destination KV pool complete. Use this counter to verify cross-instance imports. Measure latency and throughput for your workload; transfer costs can exceed the saved prefill computation.

---

### Doc Source User Guide Benchmarks (`doc/source/user_guide/benchmarks.rst`)

Model performance tests
=======================

.. versionadded:: v4.2.0
   Benchmark tests can be started from the Web UI.

Open **Performance Tests**, select a running chat model, and start a test. You can also open this page from the performance-test button on a running model. Select history records to compare results, or export a record as JSON.

.. figure:: /_static/benchmarks_setup.png
   :alt: Configure a benchmark for a running chat model.
   :width: 100%
   :align: center

   Configure a benchmark for a running chat model.

Configure the approximate input token target, maximum output tokens, comma-separated concurrency points, requests per point, warmup requests, and random seed. When a separate token-count endpoint is unavailable, calibration uses single-token chat requests and their prompt usage; these probes are excluded from measurements. Actual input and output lengths come from response usage; models may stop before the output limit. Warmup is excluded, and each concurrency point uses independent synthetic prompts. Server caches are not flushed.

Tests run sequentially in background processes on the API host and send requests through the Xinference chat API, including its replica routing. Closing the page does not stop a test. Stop a task before deleting it. Results, logs, and configuration snapshots are saved under ``XINFERENCE_HOME/benchmarks``. Restarting the API server interrupts active tasks and preserves their history.

TTFT includes reasoning output. TPOT measures decode time per output token, excluding the first token. ITL measures gaps between streamed chunks, which may contain multiple tokens. The table shows averages and tail percentiles; unavailable measurements appear as a dash. Compare the workload, actual token lengths, runner metric version, and environment snapshots before comparing performance.

With authentication enabled, these endpoints require both ``models:write`` and ``models:read``. Jobs receive a temporary credential when execution starts, restricted to the target model's chat and token-count endpoints. It stays in memory, travels to the runner over stdin, and is revoked when execution ends. Session expiry does not interrupt queued or running jobs. Disabling the account, resetting its password, or removing model-management permissions revokes job authorization. HTTP 401/403 aborts the current concurrency point without saving its metrics. Tests use live serving capacity.

The built-in ``xinference`` runner reuses the serving benchmark implementation. Runner adapters expose ``run``, ``cancel``, and ``capabilities``; records retain the runner version and raw request measurements. Additional tools can implement the adapter contract without changing task storage or the results page.

Each test displays output throughput, TTFT, and TPOT curves against concurrency. Use Latency statistic to switch the latency charts between mean, P95, and P99. Selected history records are overlaid with matching colors across charts. Points represent measurements; missing values stay blank. A notice identifies different workloads or runner metric versions.

.. figure:: /_static/benchmarks_compare.png
   :alt: Compare saved runs with throughput and P99 latency curves.
   :width: 100%
   :align: center

   Compare saved runs with throughput and P99 latency curves.

API example::

    curl -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{"model_uid":"my-model","concurrency":[1,4,16],"requests":100}' \
      http://localhost:9997/v1/benchmarks

    curl -H "Authorization: Bearer $TOKEN" \
      http://localhost:9997/v1/benchmarks

---

### Doc Source User Guide Client Api (`doc/source/user_guide/client_api.rst`)

.. _user_guide_client_api:

==========
Client API
==========

Complete Client API Reference: :ref:`reference_index`

Replica management
==================

The synchronous and asynchronous clients accept ``replica_config`` in
``launch_model`` and expose ``add_model_replica`` and
``terminate_model_replica`` for running models. See :ref:`launch` for the
request structure, placement rules, and examples.

To utilize the Client API, initiate the xinference server using the command below:

.. code-block::

    >>> xinference
    2023-10-17 16:32:21,700 xinference   24584 INFO     Xinference successfully started. Endpoint: http://127.0.0.1:9997
    2023-10-17 16:32:21,700 xinference.core.supervisor 24584 INFO     Worker 127.0.0.1:62590 has been added successfully
    2023-10-17 16:32:21,701 xinference.deploy.worker 24584 INFO     Xinference worker successfully started.

Based on the log above, the endpoint is `http://127.0.0.1:9997`. Users can connect to the xinference server through this endpoint using the Client.

Models are categorized into LLM, embedding, image, etc. We plan to introduce more model types in the future.

LLM
~~~

To list the available built-in LLM models:

.. code-block::

    >>> xinference registrations -t LLM

    Type    Name                     Language      Ability                        Is-built-in
    ------  -----------------------  ------------  -----------------------------  -------------
    LLM     baichuan                 ['en', 'zh']  ['embed', 'generate']          True
    LLM     baichuan-2               ['en', 'zh']  ['embed', 'generate']          True
    LLM     baichuan-2-chat          ['en', 'zh']  ['embed', 'generate', 'chat']  True
    ...

To initialize an LLM and chat:

Xinference Client
=================

.. code-block::

    from xinference.client import Client

    client = Client("http://localhost:9997")
    # The chatglm2 model has the capabilities of "chat" and "embed".
    model_uid = client.launch_model(model_name="glm4-chat",
                                    model_engine="llama.cpp",
                                    model_format="ggufv2",
                                    model_size_in_billions=9,
                                    quantization="Q4_K")
    model = client.get_model(model_uid)

    messages = [{"role": "user", "content": "What is the largest animal?"}]
    # If the model has "generate" capability, then you can call the
    # model.generate API.
    model.chat(
        messages,
        generate_config={"max_tokens": 1024}
    )

OpenAI Client
=============

Openai client request with the same function as before, excluding launch model. 
More details refer to: https://platform.openai.com/docs/api-reference/chat?lang=python

.. code-block::

    import openai

    # Assume that the model is already launched.
    # The api_key can't be empty, any string is OK.
    client = openai.Client(api_key="not empty", base_url="http://localhost:9997/v1")
    client.chat.completions.create(
        model=model_uid,
        messages=[
            {
                "content": "What is the largest animal?",
                "role": "user",
            }
        ],
        max_tokens=1024
    )

Cache hit token counts
======================

With vLLM, SGLang, and MLX, completion and chat completion responses expose the engine's reused prompt-token count in ``usage.prompt_tokens_details.cached_tokens`` when available. This count includes native prefix-cache reuse and Xavier cache reuse reported by the engine; it is part of ``prompt_tokens``, not an additional token count.

With P/D disaggregation, this count includes KV transferred from the prefill engine, even when that KV was computed for the current request; it does not separately measure historical prefix-cache hits on the prefill engine.

For streaming requests, set ``stream_options={"include_usage": True}`` to receive the final usage chunk with ``choices=[]``. The Responses API exposes the same count as ``usage.input_tokens_details.cached_tokens``. If the engine does not report cache hits, completion and chat completion responses omit ``prompt_tokens_details``; the Responses API defaults the count to zero.

OpenAI Client Tool Calls
========================

.. code-block::

    import openai

    tools = [
        {
            "type": "function",
            "function": {
                "name": "uber_ride",
                "description": "Find suitable ride for customers given the location, "
                "type of ride, and the amount of time the customer is "
                "willing to wait as parameters",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "loc": {
                            "type": "int",
                            "description": "Location of the starting place of the Uber ride",
                        },
                        "type": {
                            "type": "string",
                            "enum": ["plus", "comfort", "black"],
                            "description": "Types of Uber ride user is ordering",
                        },
                        "time": {
                            "type": "int",
                            "description": "The amount of time in minutes the customer is willing to wait",
                        },
                    },
                },
            },
        }
    ]

    # Assume that the model is already launched.
    # The api_key can't be empty, any string is OK.
    client = openai.Client(api_key="not empty", base_url="http://localhost:9997/v1")
    client.chat.completions.create(
        model="chatglm3",
        messages=[{"role": "user", "content": "Call me an Uber ride type 'Plus' in Berkeley at zipcode 94704 in 10 minutes"}],
        tools=tools,
    )

Output:

.. code-block::

    ChatCompletion(id='chatcmpl-ad2f383f-31c7-47d9-87b7-3abe928e629c', choices=[Choice(finish_reason='tool_calls', index=0, message=ChatCompletionMessage(content="```python\ntool_call(loc=94704, type='plus', time=10)\n```", role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_ad2f383f-31c7-47d9-87b7-3abe928e629c', function=Function(arguments='{"loc": 94704, "type": "plus", "time": 10}', name='uber_ride'), type='function')]))], created=1704687803, model='chatglm3', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=-1, prompt_tokens=-1, total_tokens=-1))

.. _openai_responses_client:

OpenAI Client Responses API
===========================

    Responses API's access address is: /v1/responses

The Responses API is stateless in Xinference. Responses are not stored, so
``previous_response_id`` and ``conversation`` are rejected: send the whole
conversation in ``input`` on every request. Text and image input, function and
custom tools, reasoning, structured output and streaming are supported. Hosted
tools such as ``web_search`` are ignored.

.. code-block::

    import openai

    client = openai.Client(api_key="not empty", base_url="http://localhost:9997/v1")
    response = client.responses.create(
        model="qwen3",
        instructions="You are a helpful assistant.",
        input="What is the largest animal?",
    )
    print(response.output_text)

Codex CLI calls custom providers through this API. Add Xinference as a provider
in ``~/.codex/config.toml``:

.. code-block:: toml

    model = "qwen3"
    model_provider = "xinference"

    [model_providers.xinference]
    name = "Xinference"
    base_url = "http://localhost:9997/v1"
    wire_api = "responses"

.. _anthropic_client:

Anthropic Client
================

See :ref:`anthropic_api` for Claude Code setup, Python SDK examples, and
supported features and limitations.

Embedding
~~~~~~~~~

To list the available built-in embedding models:

.. code-block::

    >>> xinference registrations -t embedding

    Type       Name                     Language      Dimensions  Is-built-in
    ---------  -----------------------  ----------  ------------  -------------
    embedding  bge-base-en              ['en']               768  True
    embedding  bge-base-en-v1.5         ['en']               768  True
    embedding  bge-base-zh              ['zh']               768  True
    ...

To launch an embedding model and embed text:

Xinference Client
=================

.. code-block::

    from xinference.client import Client

    client = Client("http://localhost:9997")
    # The bge-small-en-v1.5 is an embedding model, so the `model_type` needs to be specified.
    model_uid = client.launch_model(model_name="bge-small-en-v1.5", model_type="embedding")
    model = client.get_model(model_uid)

    input_text = "What is the capital of China?"
    model.create_embedding(input_text)

Output:

.. code-block::

    {'object': 'list',
     'model': 'da2a511c-6ccc-11ee-ad07-22c9969c1611-1-0',
     'data': [{'index': 0,
     'object': 'embedding',
     'embedding': [-0.014207549393177032,
        -0.01832585781812668,
        0.010556723922491074,
        ...
        -0.021243810653686523,
        -0.03009396605193615,
        0.05420297756791115]}],
     'usage': {'prompt_tokens': 37, 'total_tokens': 37}}

OpenAI Client
=============

Openai client request with the same function as before, excluding launch model. 
More details refer to: https://platform.openai.com/docs/api-reference/embeddings?lang=python

.. code-block::

    import openai

    # Assume that the model is already launched.
    # The api_key can't be empty, any string is OK.
    client = openai.Client(api_key="not empty", base_url="http://localhost:9997/v1")
    client.embeddings.create(model=model_uid, input=["What is the capital of China?"])

Output:

.. code-block::

    CreateEmbeddingResponse(data=[Embedding(embedding=[-0.014207549393177032, -0.01832585781812668, 0.010556723922491074, ..., -0.021243810653686523, -0.03009396605193615, 0.05420297756791115], index=0, object='embedding')], model='bge-small-en-v1.5-1-0', object='list', usage=Usage(prompt_tokens=37, total_tokens=37))

Image
~~~~~

To list the available built-in image models:

.. code-block::

    >>> xinference registrations -t image

    Type    Name                          Family            Is-built-in
    ------  ----------------------------  ----------------  -------------
    image   sd-turbo                      stable_diffusion  True
    image   sdxl-turbo                    stable_diffusion  True
    image   stable-diffusion-v1.5         stable_diffusion  True
    image   stable-diffusion-xl-base-1.0  stable_diffusion  True

To initiate an image model and generate an image using a text prompt:

Xinference Client
=================

.. code-block::

    from xinference.client import Client

    client = Client("http://localhost:9997")
    # The stable-diffusion-v1.5 is an image model, so the `model_type` needs to be specified.
    # Additional kwargs can be passed to AutoPipelineForText2Image.from_pretrained here.
    model_uid = client.launch_model(model_name="stable-diffusion-v1.5", model_type="image")
    model = client.get_model(model_uid)

    input_text = "an apple"
    model.text_to_image(input_text)

Output:

.. code-block::

    {'created': 1697536913,
     'data': [{'url': '/home/admin/.xinference/image/605d2f545ac74142b8031455af31ee33.jpg',
     'b64_json': None}]}

OpenAI Client
=============

Openai client request with the same function as before, excluding launch model. 
More details refer to: https://platform.openai.com/docs/api-reference/images/create?lang=python

.. code-block::

    import openai

    # Assume that the model is already launched.
    # The api_key can't be empty, any string is OK.
    client = openai.Client(api_key="not empty", base_url="http://localhost:9997/v1")
    client.images.generate(model=model_uid, prompt="an apple")


Output:

.. code-block::

    ImagesResponse(created=1704445354, data=[Image(b64_json=None, revised_prompt=None, url='/home/admin/.xinference/image/605d2f545ac74142b8031455af31ee33.jpg')])


Audio
~~~~~

To list the available built-in audio models:

.. code-block::

    >>> xinference registrations -t audio

    Type    Name               Family    Multilingual    Is-built-in
    ------  -----------------  --------  --------------  -------------
    audio   whisper-base       whisper   True            True
    audio   whisper-base.en    whisper   False           True
    audio   whisper-large-v3   whisper   True            True
    audio   whisper-medium     whisper   True            True
    audio   whisper-medium.en  whisper   False           True
    audio   whisper-tiny       whisper   True            True
    audio   whisper-tiny.en    whisper   False           True
    audio   speech_campplus_sv_zh-cn_16k-common                    campplus  False  True
    audio   speech_campplus_sv_zh_en_16k-common_advanced           campplus  True   True


To initiate an audio model and get text from an audio:

Xinference Client
=================

.. code-block::

    from xinference.client import Client

    client = Client("http://localhost:9997")
    model_uid = client.launch_model(model_name="whisper-large-v3", model_type="audio")
    model = client.get_model(model_uid)

    input_text = "an apple"
    with open("audio.mp3", "rb") as audio_file:
        model.transcriptions(audio_file.read())

Output:

.. code-block::

    {
      "text": "Imagine the wildest idea that you've ever had, and you're curious about how it might scale to something that's a 100, a 1,000 times bigger. This is a place where you can get to do that."
    }


OpenAI Client
=============

Openai client request with the same function as before.
More details refer to: https://platform.openai.com/docs/api-reference/audio/createTranscription

.. code-block::

    import openai

    # Assume that the model is already launched.
    # The api_key can't be empty, any string is OK.
    client = openai.Client(api_key="not empty", base_url="http://localhost:9997/v1")
    with open("audio.mp3", "rb") as audio_file:
        completion = client.audio.transcriptions.create(model=model_uid, file=audio_file)

Output:

.. code-block::

    Translation(text=' This list lists the airlines in Hong Kong.')


Speaker Embeddings
==================

Speaker-embedding audio models extract a fixed-length representation of speaker
identity. Xinference provides two built-in CAMPPlus models:
``speech_campplus_sv_zh-cn_16k-common`` and
``speech_campplus_sv_zh_en_16k-common_advanced``. Both return a 192-dimensional
vector through the ``speaker_embedding`` ability.

Launch a model and call it with the Xinference client:

.. code-block:: python

    from xinference.client import Client

    client = Client("http://localhost:9997")
    model_uid = client.launch_model(
        model_name="speech_campplus_sv_zh-cn_16k-common",
        model_type="audio",
    )
    model = client.get_model(model_uid)

    with open("speaker.wav", "rb") as audio_file:
        result = model.create_embedding(audio_file.read())

    print(result["dimensions"])
    embedding = result["embedding"]

The equivalent HTTP request uploads the model UID and audio file as multipart
form fields:

.. code-block:: bash

    curl -X POST 'http://localhost:9997/v1/audio/embeddings' \
      -H 'accept: application/json' \
      -F 'model=<MODEL_UID>' \
      -F '[email protected]'

The response contains one vector rather than the list-shaped response returned
by the text Embeddings API:

.. code-block:: json

    {
      "object": "embedding",
      "model": "<MODEL_UID>",
      "dimensions": 192,
      "embedding": [0.0123, -0.0456, 0.0789]
    }

Compare vectors with cosine similarity for speaker verification or speaker
identification. See :ref:`audio` for Web UI usage, input processing details,
and the complete Speaker Embedding API example.


Rerank
~~~~~~
To launch a rerank model and compute the similarity scores:

.. code-block::

    from xinference.client import Client

    client = Client("http://localhost:9997")
    model_uid = client.launch_model(model_name="bge-reranker-base", model_type="rerank")
    model = client.get_model(model_uid)

    query = "A man is eating pasta."
    corpus = [
        "A man is eating food.",
        "A man is eating a piece of bread.",
        "The girl is carrying a baby.",
        "A man is riding a horse.",
        "A woman is playing violin."
    ]
    print(model.rerank(corpus, query))

Output:

.. code-block::

    {'id': '480dca92-8910-11ee-b76a-c2c8e4cad3f5', 'results': [{'index': 0, 'relevance_score': 0.9999247789382935,
     'document': 'A man is eating food.'}, {'index': 1, 'relevance_score': 0.2564932405948639,
     'document': 'A man is eating a piece of bread.'}, {'index': 3, 'relevance_score': 3.955026841140352e-05,
     'document': 'A man is riding a horse.'}, {'index': 2, 'relevance_score': 3.742107219295576e-05,
     'document': 'The girl is carrying a baby.'}, {'index': 4, 'relevance_score': 3.739788007806055e-05,
     'document': 'A woman is playing violin.'}]}


Chat completion errors
======================

For ``/v1/chat/completions``, unexpected worker failures return a generic error message. Server logs and error events retain the original diagnostics.

Before streaming starts, system-message ordering errors on strict templates and detected vLLM multimodal per-prompt limit violations return HTTP 400. If live limit lookup is unavailable, validation is deferred to the worker.

For non-streaming requests, recognized context-length violations and known media or cache-salt validation errors return HTTP 400; rate-limit errors return HTTP 429. Messages describe the failure without exposing worker details. Unrecognized worker failures return HTTP 500.

After a streaming response starts, its HTTP status remains 200. Failures are sent in an SSE ``error`` payload. Recognized context-length and rate-limit failures retain distinct safe messages; unexpected failures use the generic message.

---

### Doc Source User Guide Continuous Batching (`doc/source/user_guide/continuous_batching.rst`)

.. _user_guide_continuous_batching:

===================
Continuous Batching
===================

Continuous batching, as a means to improve throughput during model serving, has already been implemented in inference engines like ``VLLM``.
Xinference aims to provide this optimization capability when using the transformers engine as well.

Usage
=====

LLM
---
Currently, this feature can be enabled under the following conditions:

* First, set the environment variable ``XINFERENCE_TRANSFORMERS_ENABLE_BATCHING`` to ``1`` when starting xinference. For example:

.. code-block::

    XINFERENCE_TRANSFORMERS_ENABLE_BATCHING=1 xinference-local --log-level debug


.. note::
   Since ``v0.16.0``, this feature is turned on by default and
   is no longer required to set the ``XINFERENCE_TRANSFORMERS_ENABLE_BATCHING`` environment variable.
   This environment variable has been removed.


* Then, ensure that the ``transformers`` engine is selected when launching the model. For example:

.. tabs::

  .. code-tab:: bash shell

    xinference launch -e <endpoint> --model-engine transformers -n qwen1.5-chat -s 4 -f pytorch -q none

  .. code-tab:: bash cURL

    curl -X 'POST' \
      'http://127.0.0.1:9997/v1/models' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
      "model_engine": "transformers",
      "model_name": "qwen1.5-chat",
      "model_format": "pytorch",
      "size_in_billions": 4,
      "quantization": "none"
    }'

  .. code-tab:: python

    from xinference.client import Client
    client = Client("http://127.0.0.1:9997")
    model_uid = client.launch_model(
      model_engine="transformers",
      model_name="qwen1.5-chat",
      model_format="pytorch",
      model_size_in_billions=4,
      quantization="none"
    )
    print('Model uid: ' + model_uid)


Once this feature is enabled, all requests for LLMs will be managed by continuous batching,
and the average throughput of requests made to a single model will increase.
The usage of the LLM interface remains exactly the same as before, with no differences.

Embedding
---------

With vLLM >= 0.19.0, text embedding models use the native asynchronous pooling scheduler. Configure ``max_num_seqs`` and ``max_num_batched_tokens`` at model launch; Xinference ``batch_size`` and ``batch_interval`` do not control this path. Older vLLM versions, VACC, WeMM and Qwen3-VL embeddings retain the existing Xinference batching path.

Rerank
------

With vLLM >= 0.19.0, text cross-encoder rerank models also use native asynchronous pooling. The same ``max_num_seqs`` and ``max_num_batched_tokens`` settings apply instead of Xinference batch settings. Older vLLM versions, VACC, Qwen3-VL rerankers and other scoring types retain the existing path. Ranking, ``top_n``, returned documents and token counts keep the same API behavior.

Image Model
-----------
For image-model batching, select the ``vLLM`` or ``SGLang`` engine with ``--model-engine``. SGLang supports dynamic request batching via ``batching_max_size`` and ``batching_delay_ms``; new requests wait while a batch is running. vLLM-Omni supports request batching via ``max_num_seqs`` and, on supported models, experimental step-level continuous batching with ``step_execution=true`` and ``max_num_seqs>1``. Diffusers supports ordinary image generation and multiple images per request, but does not provide Xinference-managed continuous batching. See :ref:`image` for supported models, required versions, and launch instructions.

Audio request batching
----------------------

Whisper with the ``transformers`` engine and Qwen3-TTS with the ``PyTorch`` or ``MLX`` engine collect concurrent requests into native batches within each model replica. The audio API remains unchanged.

Set ``request_batch_size`` (default ``4``) and ``batch_interval`` (default ``0.01`` seconds) at model launch. Set ``request_batch_size=1`` to disable request merging and MLX segment batching, or lower it when device memory is limited. Whisper's existing ``batch_size`` still controls the pipeline's audio-chunk batch size.

.. code-block:: bash

    xinference launch --model-name whisper-large-v3 --model-type audio --model-engine transformers --request_batch_size 4 --batch_interval 0.01
    xinference launch --model-name Qwen3-TTS-12Hz-0.6B-CustomVoice --model-type audio --model-engine MLX --request_batch_size 4 --batch_interval 0.01

Only requests with compatible inference settings are merged. Qwen3-TTS requests with an explicit seed run separately. MLX batching also requires the same language and reference audio; unsupported batch options use the existing single-request path. Long MLX Qwen3-TTS inputs can batch text segments while preserving their order and pauses. Streaming support is unchanged.

Abort your request
==================
For LLMs using Xinference's continuous batching, you can abort requests that are in the process of inference.

#. First, add ``request_id`` option in ``generate_config``. For example:

.. code-block:: bash

    from xinference.client import Client
    client = Client("http://127.0.0.1:9997")
    model = client.get_model("<model_uid>")
    model.chat([{"role": "user", "content": ""}], generate_config={"request_id": "<your_unique_request_id>"})

#. Then, abort the request using the ``request_id`` you have set. For example:

.. code-block:: bash

    from xinference.client import Client
    client = Client("http://127.0.0.1:9997")
    client.abort_request("<model_uid>", "<your_unique_request_id>")

Note that if your request has already finished, aborting the request will be a no-op.

Note
====

* Currently, for ``LLM`` models, this feature only supports the ``generate``, ``chat``, ``tool call`` and ``vision`` tasks.

* For ``vision`` tasks, currently only ``qwen2-vl-instruct``, ``qwen2.5-vl-instruct``, ``QvQ-72B-Preview``, ``glm-4v`` and ``MiniCPM-V-2.6`` (only for image tasks) models are supported. More models will be supported in the future. Please let us know your requirements.

* If using GPU inference, this method will consume more GPU memory. Please be cautious when increasing the number of concurrent requests to the same model.
  The ``launch_model`` interface provides the ``max_num_seqs`` parameter to adjust the concurrency level, with a default value of ``16``.

---

### Doc Source User Guide Distributed Inference (`doc/source/user_guide/distributed_inference.rst`)

.. _distributed_inference:

#####################
Distributed Inference
#####################
Some language models including **DeepSeek V3**, **DeepSeek R1**, etc are too large to fit into GPus
on a single machine, Xinference supported running these models across multiple machines.

.. versionadded:: v1.3.0

*****************
Supported Engines
*****************
Now, Xinference supported below engines to run models across workers.

* :ref:`SGLang <sglang_backend>` (supported in v1.3.0)
* :ref:`vLLM <vllm_backend>` (supported in v1.4.1)
* :ref:`MLX <mlx_backend>` (supported in v1.7.1), MLX distributed currently does not support all models.
  The following model types are supported at this time. If you have additional requirements,
  feel free to submit a GitHub issue at `https://github.com/xorbitsai/inference/issues <https://github.com/xorbitsai/inference/issues>`_ to request support.

  - DeepSeek v3 and R1
  - Qwen2.5-instruct and the models have the same model architectures.
  - Qwen3 and the models have the same model architectures.
  - Qwen3-moe and the models have the same model architectures.


*****
Usage
*****
First you need at least 2 workers to support distributed inference.
Refer to :ref:`running Xinference in cluster <distributed_getting_started>`
to create a Xinference cluster including supervisor and workers.

vLLM V1 (v0.11.1+) supports tensor parallelism (TP) and pipeline parallelism (PP)
through Xinference's xoscar executor. For a model spanning workers, the defaults
are ``tensor_parallel_size`` = GPUs per worker and ``pipeline_parallel_size`` = worker count.

For example, two workers with four GPUs each use ``--n-worker 2 --n-gpu 4``,
``tensor_parallel_size=4`` and ``pipeline_parallel_size=2``. Explicit TP/PP settings
must satisfy ``TP * PP = total allocated GPUs``. Each worker must allocate the same
GPU count, and the model architecture must support vLLM PP.

This PP deployment does not require a Ray cluster. Xavier V1 and native NIXL P/D
separation currently require ``TP=1`` and ``PP=1`` and cannot be combined with this configuration.

Then if are using web UI, choose expected machines for ``worker count`` in the optional configurations,
if you are using command line, add ``--n-worker <machine number>`` when launching a model.
The model will be launched across multiple workers accordingly.

.. raw:: html

    

``GPU count`` on web UI, or ``--n-gpu`` for command line now mean GPUs count per worker if you are using distributed inference.

---

### Doc Source User Guide Index (`doc/source/user_guide/index.rst`)

.. _user_guide_index:

==========
User Guide
==========

.. toctree::
   :maxdepth: 2
   :caption: Model usage

   launch
   backends
   client_api
   realtime
   anthropic
   SD WebUI API compatibility <sd_webui>

.. toctree::
   :maxdepth: 2
   :caption: Deployment

   using_docker_swarm
   zero_downtime_upgrade

.. toctree::
   :maxdepth: 2
   :caption: Operations and security

   Metrics <metrics>
   auth_system
   oidc_sso
   audit_security

.. toctree::
   :maxdepth: 2
   :caption: Advanced performance

   benchmarks
   continuous_batching
   speculative_decoding
   distributed_inference
   Xavier <vllm_enhancement>
   PD separation 

---

### Doc Source User Guide Launch (`doc/source/user_guide/launch.rst`)

.. _launch:

============================
Model Launching Instructions
============================

This document aims to provide a functional overview of model launching.

On **Running Models**, choose **Clear failed record** to remove a failed launch after its runtime has been cleaned up. You can also use ``DELETE /v1/models/{model_uid}``. This clears the status record and keeps cached model files. With authentication enabled, the operation requires ``models:write`` permission.

An initial launch remains cancellable until all its replicas finish, even when some are already ready. **Cancel launch** on **Running Models** calls ``POST /v1/models/{model_uid}/cancel`` to stop pending work and remove replicas created by that launch. ``GET /v1/models/instances`` reports ``can_cancel_launch`` for this operation. Scale-up of an established model does not enable initial-launch cancellation.

.. note::

   The LMDeploy backend has been removed because its Xinference integration is
   no longer maintained. For InternVL models, select vLLM or Transformers instead.

Updating the model catalog
==========================

On the **Launch Model** page, choose the model type beside **Update**. **Update** refreshes the whole category. Choose **Update one model** and enter the exact Hub model name (or world family name) to update just that model, including a model not yet listed. Other catalog entries are preserved. This updates shared metadata without downloading weights or restarting running models. With authentication enabled, both actions require administrator or ``models:register`` permission.

Model favorites
===============

Stars on the **Launch Model** page are saved for the current account and survive browser changes and server restarts. Favorites are identified by model name across model categories. Reading or changing favorites requires ``models:list`` permission. Without authentication, all visitors share one collection. With authentication enabled, changes require a user session; API keys can only read.

Favorites use a ``model_favorites`` table in the selected authentication database. With the default SQLite backend, this is ``<XINFERENCE_HOME>/auth/auth.db`` (configured by ``XINFERENCE_AUTH_DB_PATH``). Without authentication, the same path stores the shared collection and only the favorites table is initialized.

Old browser favorites are not imported automatically because they have no account identity. Choose **Import browser favorites** to add them to the current account. The browser copy is retained if any write fails or the account changes during import; retrying does not create duplicates.

``GET /v1/model_favorites`` returns a list of model names for the authenticated account. ``PUT`` and ``DELETE`` on the same path accept ``{"model_name": "org/model"}`` to add or remove one name. Both operations are idempotent. The server derives the owner from authentication; clients cannot choose another owner.

.. _launch_recommendations:

LLM launch recommendations
==========================

.. versionadded:: v3.5.0

In an LLM's deployment dialog, use the launch recommendation button to get a
suggested configuration, then review it before deploying. The button is available
for LLMs, embedding, rerank, and audio models. Recommendations currently target a single replica.
Custom model paths and custom engine parameters require manual configuration.

How recommendations are chosen
------------------------------

Click **Recommend configuration** in the upper-right corner of the launch
dialog to fill in the engine, format, size, and quantization together. You can
still change these fields before clicking **Deploy**. The recommendation also
selects a worker, so the configuration is tied to the machine checked for it.

Xinference starts with your choices. If you have selected an 8B model, it looks
for configurations of that size; it will not switch to a smaller model. Worker,
GPU, and virtual environment settings also constrain the search. It checks each
worker separately and removes combinations that fail the compatibility checks.
If none remain, it leaves your form unchanged so you can adjust the settings.

If memory data is available, Xinference prefers the largest size estimated to
fit. Otherwise it falls back to the smallest unverified size. Within that size,
installed engines are preferred; virtual-environment candidates remain eligible.

Next comes the platform's default engine order. With the earlier conditions
equal, MLX comes first on Apple Silicon, vLLM followed by SGLang on Linux with
CUDA, and llama.cpp on CPU. Within an engine, common 4-bit quantizations such as
Q4_K_M and Int4 are preferred when supported. If choices are still tied, weights
already cached on that worker get priority. A cached model does not override
your selected size or the earlier preferences.

Estimates use offline architecture facts and checkpoint sizes when available.
MoE weights include all resident experts. Full-attention KV cache, Gated DeltaNet
recurrent state and workspace are counted separately. MLX, Transformers, vLLM,
SGLang and llama.cpp share this accounting with different workspace bounds.
Unknown layouts, including MLA, remain unverified.

Defaults are one sequence, the model's registered maximum context, FP16 KV cache, no image tokens and
no retained prefix cache. The budget is 80% of currently free memory: the lowest
free memory among eligible GPUs, or available system memory on CPU and Apple
Silicon. The default vLLM GPU reservation is checked separately when total memory
is known; SGLang budgets from memory available at initialization. Multi-GPU
requests keep compatibility-only selection.
Known oversized candidates are excluded. Workspace bounds are estimates, not
measured peaks. Engine preallocation is not modeled beyond the separate vLLM
default reservation check; custom engine settings require manual configuration.
No resources are reserved.

Using the API
-------------

``POST /v1/models/recommend`` is read-only. Supply ``model_name`` and
``model_type="LLM"``; the optional ``constraints`` object accepts
``model_size_in_billions``, ``worker_ip``, ``enable_virtual_env``, ``n_gpu``, and
``gpu_idx``. LLM constraints also accept ``context_length``. Supplied constraints
are hard requirements, never silently relaxed.

LLM constraints also accept positive integers ``context_length`` (text input plus
output tokens) and ``concurrency``, and non-negative integers ``vision_tokens``
(encoded image tokens per sequence) and ``prefix_cache_bytes`` (additional retained
cache across the instance). Omitting ``context_length`` uses each candidate's
registered maximum context, including encoded image tokens. Models without a
known maximum remain unverified. Recommended vLLM ``max_model_len`` and SGLang
``context_length`` equal text plus vision tokens per sequence. llama.cpp uses
``n_parallel=concurrency`` and ``n_ctx=(context_length + vision_tokens) * concurrency``
because its context is divided across slots. The Web UI applies these advanced
launch parameters. Other workload fields affect estimates rather than request
limits. Older workers that do not support workload options are retried without
them and remain unverified, with a version warning. Image workloads require vision metadata.
For multiple images, ``vision_tokens_per_image`` specifies their maximum encoded
token count; zero assumes all image tokens belong to one image. Cached states grow
with concurrency, while model weights are counted once. CUDA profiles for vLLM and SGLang share a
16,384-token prefill workspace profile with fused attention and sampled-row logits.

Recommendation configs expose ``context_length`` as the total per-request budget,
including encoded image tokens. llama.cpp also returns ``n_parallel`` to match the
estimated concurrency; the launch layer maps the unified budget to each engine.

.. code-block:: bash

    curl -X POST http://127.0.0.1:9997/v1/models/recommend \
      -H 'Content-Type: application/json' \
      -d '{
        "model_name": "qwen2.5-instruct",
        "model_type": "LLM",
        "constraints": {"model_size_in_billions": "7", "n_gpu": 1}
      }'

The response ``status`` is ``recommended`` or ``no_recommendation``. On success,
``config`` contains launch-compatible fields such as ``model_engine``,
``model_format``, ``model_size_in_billions``, ``quantization``, and ``worker_ip``.
``reasons`` and ``warnings`` contain entries with ``code`` and ``message`` fields.
If no configuration satisfies the constraints, review the explanation and adjust
the constraints or configure the launch manually.

A recommendation does not guarantee sufficient memory or a successful launch.
It reserves no resources, downloads no model files, and installs no dependencies.
Existing launch behavior is unchanged; deployment remains a separate action.

``config`` also includes the effective
``enable_virtual_env`` and any supplied ``n_gpu`` and ``gpu_idx`` constraints.
``n_gpu=null`` requests CPU placement, while ``n_gpu="auto"`` uses the existing
launch allocation policy rather than calculating the required GPU count.

Both ``recommended`` and ``no_recommendation`` return HTTP 200. An unknown model
returns HTTP 404; invalid requests return HTTP 422. Reason and warning codes use
lower snake_case. The ``memory_not_verified`` warning is always included.

Unified LLM context length
==========================

Set ``context_length`` in the launch API or use ``--context-length`` in the CLI
to choose the input plus output token limit per request. In the Web UI, use the
context slider or enter an exact integer; **Model default** clears the override.
The slider starts at the model's native context length. For models and engines
with a registered YaRN recipe, its upper bound includes the verified extension
range. Hover over the question mark beside **Context length** to see the native
length, extension limit, and whether automatic YaRN is supported by the selected
engine. Context selections are preserved in configuration history and passed to
the recommendation API.

The launch layer maps this setting to vLLM's ``max_model_len``, SGLang,
Transformers and MLX's ``context_length``, and
llama.cpp's ``n_ctx=context_length*n_parallel`` total slot capacity. Existing engine
parameters still work; contradictory unified and native values are rejected.
Omitting the setting preserves engine defaults.

Qwen3.5, Qwen3.6, and Qwen3.8 have registered automatic YaRN recipes for vLLM and
SGLang; Qwen3.8-Flash-Next currently supports vLLM only. Selecting a length within
the native window leaves RoPE unchanged. Selecting a longer length enables YaRN
at model launch, with the scaling factor rounded up from the requested length
divided by the native length (for example, 524,288 tokens uses a factor of 2 and
1,000,000 tokens uses a factor of 4 for a 262,144-token model). The required
environment flag is applied only to the model process. The automatic limit is
1,010,000 tokens for Qwen3.5, Qwen3.6, and Qwen3.8-Max, and 1,000,000 tokens for
Qwen3.8-27B and Qwen3.8-Flash-Next. Explicit RoPE overrides and already-scaled
checkpoint configurations take precedence. Models without a registered recipe
retain their existing engine behavior. Changing the selected length requires
restarting the model; the engine does not switch scaling per request.

For models with a registered YaRN recipe, weight-cache reload cannot increase
the current context length. Deploy a new model to increase it so the checkpoint
and RoPE settings are checked again.

With an explicit context limit, MLX rejects prompts that leave no room for output
and caps ``max_tokens`` to the remaining budget. Cached prompt tokens and
processed image tokens count toward the input limit. Non-streaming requests that
exceed the input limit return HTTP 400; streaming requests report an SSE error
without starting generation.

.. code-block:: bash

    xinference launch --model-name qwen3.5 --model-engine vLLM \
      --size-in-billions 9 --model-format awq --quantization Int4 \
      --context-length 32768

To launch Qwen3.8-27B with automatic YaRN extension to one million tokens:

.. code-block:: bash

    xinference launch --model-name qwen3.8 --model-engine vLLM \
      --size-in-billions 27 --model-format pytorch --quantization none \
      --context-length 1000000

Recommendations for other model types
======================================

.. versionadded:: v3.5.0

Embedding, rerank, and audio models use the same **Recommend configuration** button and ``POST /v1/models/recommend`` endpoint; set ``model_type`` to ``embedding``, ``rerank``, or ``audio``. The recommendation fills only the fields supported by that type, without selecting a model size. Embedding and rerank prefer an available sentence-transformers engine; audio prefers an available MLX engine on Apple Silicon, otherwise its standard Transformers or PyTorch backend. Installed engines come before engines that need a virtual environment, and unquantized variants are preferred within an engine. Unlike LLM recommendations, these types do not use cached weights as a tie-breaker. Worker, GPU, and virtual environment constraints still apply; ``model_size_in_billions`` is rejected for these types. A recommendation remains a starting configuration, not a memory-fit guarantee. Image and video recommendations are not yet supported.

Download without launching
==========================

.. versionadded:: v3.4.0

Open a model's deployment dialog and choose **Download only** beside the
**Deploy** button. It downloads the same model artifacts used by deployment,
but does not reserve GPUs, create model subprocesses, install a virtual
environment, or load a model into memory.

The REST endpoint is ``POST /v1/cache/models``. Supply ``cache_uid`` when the
client needs to query progress or cancel the operation while the POST request
is still running.

.. code-block:: bash

    curl -X POST http://127.0.0.1:9997/v1/cache/models \
      -H 'Content-Type: application/json' \
      -d '{
        "cache_uid": "download-qwen",
        "model_name": "qwen2.5-instruct",
        "model_type": "LLM",
        "model_engine": "transformers",
        "model_format": "pytorch",
        "model_size_in_billions": "0_5",
        "quantization": "none"
      }'

Progress and cancellation use the same ``cache_uid``:

.. code-block:: bash

    curl http://127.0.0.1:9997/v1/cache/models/download-qwen/progress
    curl -X POST http://127.0.0.1:9997/v1/cache/models/download-qwen/cancel

Cache-only downloads can also be paused and resumed:

.. code-block:: bash

    curl -X POST http://127.0.0.1:9997/v1/downloads/download-qwen/pause
    curl -X POST http://127.0.0.1:9997/v1/downloads/download-qwen/resume

Xinference persists unfinished cache-only download tasks. If the supervisor or
worker stops unexpectedly, an active task is shown as ``interrupted`` after
restart and can be resumed with the same endpoint. Resume reuses files already
present in the model hub cache. Byte-range continuation of a partially written
file depends on the selected hub client; otherwise that incomplete file is
downloaded again.

``GET /v1/downloads`` lists active, paused, interrupted, and failed downloads.
Pause and resume apply to cache-only downloads. A model launch that happens to
be downloading still uses the launch cancellation endpoint because pausing a
launch would also need to preserve its runtime allocation state.

In a distributed deployment, ``worker_ip`` can select the worker whose local
cache receives the files. If it is omitted, the supervisor selects one worker.

``GET /v1/cache/models`` includes ``model_type`` and ``model_engine`` for each cached model. The engine list reports candidates discovered on the worker that owns its files, including that worker's virtual environment setting. Rerank and world cache versions identify only the model name, so their candidates are not checked against a cached format. ``null`` means the metadata is unknown or discovery failed; an empty list means no matching engine is currently discoverable. Listing caches does not download weights, prepare environments, or launch models.

Replica
=======

During deployment, the progress endpoint ``GET /v1/models/{model_uid}/progress``
reports a stage for each replica. After model files finish downloading, a replica
can report ``waiting_for_dependencies`` while another launch prepares its shared
virtual environment, then ``installing_dependencies`` while packages are being
installed, and ``loading`` after the environment is ready.

The Web UI shows each replica's file download details inside its replica card.

During dependency installation, each replica's progress includes the number of
packages planned for the main install and the number already present in its
virtual environment. A replica waiting for a shared environment has no install
count of its own. If the installer cannot resolve a reliable plan, the count is
omitted.

The Web UI keeps the dependency installation result visible while a replica is
loading. When no dependency installation is needed, the replica card says so.

Replicas specify the number of model instances to load. For example, if you have two GPUs and each can host one replica of the model,
you can set the replica count to 2. This way, two identical instances of the model will be distributed across the two GPUs.
Xinference automatically load-balances requests to ensure even distribution across multiple GPUs.
Meanwhile, users see it as a single model, which greatly improves overall resource utilization.

Traditional Multi-Instance Deployment：

When you have multiple GPU cards, each capable of hosting one model instance, you can set the number of instances equal to the number of GPUs. For example:

- 2 GPUs, 2 instances: Each GPU runs one model instance
- 4 GPUs, 4 instances: Each GPU runs one model instance

.. versionadded:: v1.15.0

Introduce a new environment variable:

.. code-block:: bash

    XINFERENCE_ALLOW_MULTI_REPLICA_PER_GPU

Control whether to enable the single GPU multi-copy feature
Default value: 1

New Feature: Smart Replica Deployment

1. Single GPU Multi-Replica

New Support: Run multiple model replicas even with just one GPU.

- Scenario: You have 1 GPU with sufficient VRAM
- Configuration: Replica Count = 3, GPU Count = 1
- Result: 3 model instances running on the same GPU, sharing GPU resources

2. Hybrid GPU Allocation

Smart Allocation: Number of replicas may differ from GPU count; system intelligently distributes

- Scenario: You have 2 GPUs and need 3 replicas
- Configuration: Replicas=3, GPUs=2
- Result: GPU0 runs 2 instances, GPU1 runs 1 instance

Per-replica placement
---------------------

For distributed deployments, ``replica_config`` can pin every replica to a
specific worker and optional GPU indexes. Worker addresses must use the full
registered ``IP:port`` value. The number of entries must equal ``replica`` and
each entry currently supports exactly one worker.

.. code-block:: python

    from xinference.client import Client

    client = Client("http://localhost:9997")
    model_uid = client.launch_model(
        model_name="qwen2.5-instruct",
        model_engine="vllm",
        replica=2,
        replica_config=[
            {
                "replica_uid": "primary",
                "devices": [
                    {
                        "worker_ip": "192.168.1.10:9978",
                        "n_gpu": 1,
                        "gpu_idx": [0],
                    }
                ],
            },
            {
                "replica_uid": "secondary",
                "devices": [
                    {
                        "worker_ip": "192.168.1.11:9978",
                        "n_gpu": 1,
                        "gpu_idx": [0],
                    }
                ],
            },
        ],
    )

The same placement can be supplied to ``xinference launch`` as a JSON array:

.. code-block:: bash

    xinference launch \
      --model-name qwen2.5-instruct \
      --model-engine vLLM \
      --replica 2 \
      --replica-config '[{"replica_uid":"primary","devices":[{"worker_ip":"192.168.1.10:9978","n_gpu":1,"gpu_idx":[0]}]},{"replica_uid":"secondary","devices":[{"worker_ip":"192.168.1.11:9978","n_gpu":1,"gpu_idx":[0]}]}]'

``--replica_config`` remains available as a compatibility alias, but
``--replica-config`` is recommended. If ``--replica`` is omitted, the CLI
derives it from the number of array entries. If it is supplied explicitly, it
must equal that number.

``replica_config`` is mutually exclusive with the model-level ``worker_ip``,
``n_gpu``, and ``gpu_idx`` arguments, and with ``n_worker > 1``. The CLI
therefore rejects ``--replica-config`` together with ``--worker-ip``,
``--gpu-idx``, a non-``auto`` ``--n-gpu``, or ``--n-worker`` greater than 1.
Each replica currently targets exactly one worker. If ``replica_uid`` is
omitted, Xinference assigns the stable default
``{model_uid}-{replica_index}`` (for example, ``my-model-0``). Omit ``gpu_idx``
and use ``n_gpu="auto"`` to let the selected worker allocate GPUs
automatically.

Running models can be scaled by one or more replicas in a single operation.
The new replicas reuse the existing launch configuration by default. You can
override their model engine and device allocation without changing existing
replicas. Placement is optional; if omitted, the supervisor selects a worker
automatically.

.. code-block:: python

    result = client.add_model_replica(model_uid)

    scale_result = client.add_model_replica(
        model_uid,
        replica=2,
        model_engine="vllm",
        n_gpu=1,
    )

    result = client.add_model_replica(
        model_uid,
        replica_config={
            "replica_uid": "burst-capacity",
            "devices": [
                {
                    "worker_ip": "192.168.1.12:9978",
                    "n_gpu": 1,
                    "gpu_idx": [1],
                }
            ],
        },
    )

    remaining = client.terminate_model_replica(
        model_uid, replica_id=result["replica_id"]
    )

For multiple replicas, ``replica_config`` may also be a list with one placement
entry per new replica. If any replica in a multi-replica scale-up fails to
launch, Xinference rolls back the replicas created by that operation.

Scale-up is not supported for Xavier-distributed models or models launched with
``n_worker > 1``. Deleting the last replica terminates the running model.

GPU Allocation Strategy
=======================

The current policy is *Idle First*: The scheduler always attempts to assign replicas to the least utilized GPU. Use the ``XINFERENCE_LAUNCH_STRATEGY`` parameter to choose launch strategy.

Set Environment Variables
=========================

.. versionadded:: v1.8.1

Sometimes, we want to specify environment variables for a particular model at runtime.
Since v1.8.1, Xinference provides the capability to configure these individually without needing to set them before starting Xinference.

For Web UI.

.. raw:: html

    

When using the command line, use ``--env`` to specify an environment variable.

Example usage:

.. code-block:: bash

  xinference launch xxx --env A 0 --env B 1

Take vLLM as an example: it has versions V1 and V0, and by default, it automatically determines which version to use.
If you want to force the use of V0 by setting ``VLLM_USE_V1=0`` when launching a model, you can specify this during model launching.

Configuring Model Virtual Environment
=====================================

.. versionadded:: v1.8.1

For this part, please refer to :ref:`toggling virtual environments and customizing dependencies <model_launching_virtualenv>`.

Batching / Continuous Batching
==============================

Xinference supports batching for higher throughput. For LLMs on the ``transformers`` engine,
continuous batching is available and can be enabled via environment variables at launch time.

Key settings:

- ``XINFERENCE_BATCH_SIZE`` and ``XINFERENCE_BATCH_INTERVAL`` for general batching behavior.

Example (LLM, transformers):

.. code-block:: bash

  XINFERENCE_BATCH_SIZE=32 XINFERENCE_BATCH_INTERVAL=0.003 xinference-local --log-level debug
  xinference launch -e <endpoint> --model-engine transformers -n qwen1.5-chat -s 4 -f pytorch -q none

For image and video models, batching depends on the selected native engine and model. Diffusers does not use a Xinference step-level scheduler. See :ref:`image` and :ref:`video`.

For detailed behavior, supported models, and aborting requests, see
:ref:`Continuous Batching `.

Thinking Mode
=============

Some hybrid reasoning models (for example, Qwen3) support an optional *thinking mode*.
You can enable this at launch time via ``--enable-thinking``.

Example usage:

.. code-block:: bash

  xinference launch -n qwen3-xxx --model-engine vllm --enable-thinking

Launch Configuration History
============================

After a model is launched successfully from the Web UI, Xinference stores its
launch configuration in the Supervisor launch-history database and keeps a
user-scoped browser cache for fast access and temporary network failures. When
the deployment dialog is opened again, the newest server-backed configuration
is restored unless the user has already started editing the form.

The configuration history dialog can remove records owned by the current user.
Records used by automatic startup are protected from deletion. In authenticated
deployments, browser caches are separated by the username in the access token;
configuration data owned by another user is not placed in that cache.

The history dialog also lists configurations owned by the current user for
other model names. These records are never applied automatically. After the
user confirms **Use as Template**, Xinference keeps the current model name,
removes the previous model UID and history or automatic-startup metadata, and
copies the remaining launch options into the form without starting a model.

.. _launch_weight_cache_reload:

.. _reconfigure-without-reloading-weights:

Reconfigure and upgrade running models
======================================

.. versionadded:: v4.0.0

Enable ``enable_weight_cache=true`` when launching a model, or select **Keep GPU weights for reload** in the deployment dialog. This opt-in feature requires vLLM 0.31.0+ or SGLang 0.5.21+, Linux with CUDA or ROCm, one worker, one replica, and fixed tensor parallelism. It is disabled by default, even on supported engine versions.

.. figure:: /_static/keep_weight_for_reload.png
   :alt: Enable Keep GPU weights for reload before deploying the model.
   :width: 560px
   :align: center

   Enable Keep GPU weights for reload before deploying the model.

.. figure:: /_static/reconfigure_in_running.png
   :alt: Open model reconfiguration from Running Models.
   :width: 100%
   :align: center

   Open model reconfiguration from Running Models.

.. figure:: /_static/reconfigure_in_detail.png
   :alt: Open Reconfigure / upgrade model from the model's detail view.
   :width: 100%
   :align: center

   Open Reconfigure / upgrade model from the model's detail view.

With retained weights enabled, Xinference prepares a replacement model process while the current engine keeps serving. The replacement shares compatible cached weights and builds its own KV cache and CUDA graphs. New requests switch only after loading and a health check succeed; existing requests, including streams, finish on their original engine. Failed preparation leaves the current engine serving. The model UID and GPU reservation stay the same. Terminating the model releases its processes and cached weights.

``POST /v1/models/{model_uid}/reload`` accepts a partial ``model_config`` object and optional ``python_executable`` and ``drain_timeout`` fields. It returns HTTP 202 with an ``operation_id``. Poll ``GET /v1/models/{model_uid}/reload`` until ``status`` is ``ready`` or ``error``. On error, ``restored=true`` means the previous engine is serving. Query ``GET /v1/models/{model_uid}/reload/config`` for supported parameters, current values, engine version and Python executable. Invalid patches return HTTP 400; a reload already in progress returns HTTP 409. ``drain_timeout`` controls the wait for parser-only updates (default 300 seconds, maximum 3600); engine replacements never force existing requests to stop.

Both engines need runtime memory during the switch: shared weights do not share KV cache or CUDA graphs. Reserve enough GPU memory for both engines. Before starting a replacement, Xinference checks free memory while preserving the current engine's configured budget, and rejects insufficient memory or unavailable driver process accounting. Both Xinference environments, and the version that originally launched the model, must support seamless model replacement.

Insufficient memory never triggers an in-place rebuild. Even models launched with the default 0.9 memory fraction need enough spare memory for replacement; otherwise reload fails while inference continues on the current engine.

For consecutive upgrades, ``drain_timeout`` also bounds waiting for previous engine retirement. A timeout leaves current requests running; a later upgrade retries failed retirement.

When authentication is enabled, selecting ``python_executable`` requires administrator access. The current interpreter path is only included in reload metadata for administrators.

To change the model environment, set ``python_executable`` to an installed absolute Python path on the Worker machine. The target must already contain Xinference and the model engine; packages are not installed by reload. Compatible engine and PyTorch versions reuse the retained weights. Different versions use a separate weight cache and need additional GPU memory; upstream compatibility checks remain enabled. In the Web UI, use **Reconfigure / upgrade model** in the running model's detail view.

.. code-block:: python

    from xinference.client import Client

    client = Client("http://127.0.0.1:9997")
    operation = client.reload_model(
        "my-model", {}, python_executable="/opt/xinference-new/bin/python"
    )

.. code-block:: bash

    xinference reload --model-uid my-model \
      --python-executable /opt/xinference-new/bin/python

vLLM supports changing ``max_num_seqs``, ``max_num_batched_tokens``, ``max_model_len``, ``gpu_memory_utilization`` and ``enforce_eager``.

For reasoning-capable models, both engines also accept ``reasoning_content`` through the same reload API. This changes only the Xinference reasoning parser and leaves the engine and GPU weights intact. It works without ``enable_weight_cache`` or the minimum engine versions required for weight retention. Existing requests, including streams, finish before the new setting is applied; new requests wait until the parser change is applied. The ``hot_parameters`` field in reload metadata lists these settings; ``enabled`` still indicates launch-time weight retention. Hot updates also require one worker and one replica. ``weights_reused`` is true only when an engine rebuild uses retained weights.

For vLLM vision models launched with weight retention, ``limit_mm_per_prompt`` can change a positive image count to another positive count, for example ``{"image": 8}``. Unspecified modality limits are preserved in the engine, recovery and automatic-startup configuration. The engine is rebuilt to apply multimodal memory profiling and cache budgets while reusing the retained weights. Setting the image count to zero or enabling images from an initially disabled encoder is rejected; limits for other modalities cannot change through this operation.

SGLang supports changing ``max_running_requests``, ``max_prefill_tokens``, ``context_length``, ``chunked_prefill_size``, ``mem_fraction_static``, ``disable_cuda_graph``, ``cuda_graph_max_bs_decode``, ``cuda_graph_max_bs_prefill`` and ``schedule_conservativeness``.

For SGLang vision models launched with weight retention, ``limit_mm_data_per_request`` accepts a partial image-count patch, for example ``{"image": 8}``. The engine is rebuilt with retained weights so the new tokenizer process applies the limit. Other modality limits are preserved in the engine, recovery and automatic-startup configuration. An initially unlimited or zero image count can become a positive count because this setting only validates request counts; it does not disable the vision encoder. New image limits must be positive integers. Returning to an unlimited or zero image count requires relaunching the model. With an initial zero image count, Xinference skips SGLang's default image warmup.

Parallelism, GPU placement, model path, precision, quantization and weight-layout options cannot change through reload. Pipeline/data parallelism, multi-worker deployments, LoRA, speculative decoding, weight offloading and PD deployments are not supported by this integration yet. Quantized models must pass the upstream IPC compatibility checks. For vLLM, multi-GPU caching uses its native multiprocessing executor. In the running-model detail page, choose **Reconfigure / upgrade model** to edit the supported values and follow progress.

.. code-block:: python

    from xinference.client import Client

    client = Client("http://127.0.0.1:9997")
    uid = client.launch_model(
        model_name="qwen2.5-instruct", model_engine="vLLM",
        model_size_in_billions="0_5", model_format="pytorch",
        quantization="none", n_gpu=1, enable_weight_cache=True,
        gpu_memory_utilization=0.4,
    )
    job = client.reload_model(uid, {"max_num_seqs": 32})
    print(job["operation_id"])
    print(client.get_model_reload_status(uid))

.. code-block:: bash

    xinference reload --model-uid MODEL_UID --model-config '{"max_num_seqs": 32}'
    curl http://127.0.0.1:9997/v1/models/MODEL_UID/reload

To change reasoning parsing without restarting the engine:

.. code-block:: python

    job = client.reload_model(uid, {"reasoning_content": True})

For a vision model launched with weight retention and a positive image limit:

.. code-block:: python

    job = client.reload_model(uid, {"limit_mm_per_prompt": {"image": 8}})

To change the image count of a SGLang vision model launched with weight retention:

.. code-block:: python

    job = client.reload_model(uid, {"limit_mm_data_per_request": {"image": 8}})

---

### Doc Source User Guide Metrics (`doc/source/user_guide/metrics.rst`)

.. _metrics:

==================
Metrics
==================

There are two types of metrics exporters in an Xinference cluster:

- Supervisor metrics exporter at `<endpoint>/metrics`, e.g. `http://127.0.0.1:9997/metrics`.
- Worker metrics exporter at each worker node, the exporter host and port can be set by `--metrics-exporter-host` and `--metrics-exporter-port` options in `xinference-local` or `xinference-worker` command.

Supervisor Metrics
^^^^^^^^^^^^^^^^^^



- **exceptions_total_counter** (counter): Total number of requested which generated an exception

- **requests_total_counter** (counter): Total number of requests received

- **responses_total_counter** (counter): Total number of responses sent

- **status_codes_counter** (counter): Total number of response status codes



Worker Metrics
^^^^^^^^^^^^^^



- **xinference:generate_tokens_per_s** (gauge): Generate throughput in tokens/s.

- **xinference:input_tokens_total_counter** (counter): Total number of input tokens.

- **xinference:output_tokens_total_counter** (counter): Total number of output tokens.

- **xinference:time_to_first_token_ms** (gauge): First token latency in ms.

---

### Doc Source User Guide Oidc Sso (`doc/source/user_guide/oidc_sso.rst`)

.. _user_guide_oidc:

======================
OIDC Single Sign-On
======================

Xinference can authenticate users against an external OpenID Connect (OIDC)
identity provider such as `Keycloak <https://www.keycloak.org/>`_, in
addition to the built-in username/password login. SSO users are provisioned
automatically on first login and managed like any other user afterwards.

OIDC requires the database-backed authentication system to be enabled (the
default) — see :ref:`user_guide_auth_system`.

Configuration
=============

Enable OIDC by setting the following environment variables on the process
that runs the RESTful API:

.. list-table::
   :header-rows: 1

   * - Environment variable
     - Meaning
   * - ``XINFERENCE_OIDC_ENABLED``
     - Set to ``1`` / ``true`` / ``yes`` to enable OIDC. Default: disabled.
   * - ``XINFERENCE_OIDC_ISSUER``
     - Issuer URL of your identity provider, e.g.
       ``https://keycloak.example.com/realms/myrealm``. Xinference discovers
       the provider's endpoints from
       ``/.well-known/openid-configuration``.
   * - ``XINFERENCE_OIDC_CLIENT_ID``
     - OAuth client ID registered at the provider.
   * - ``XINFERENCE_OIDC_CLIENT_SECRET``
     - OAuth client secret (a *confidential* client is required).
   * - ``XINFERENCE_OIDC_REDIRECT_URI``
     - Callback URL pointing back at Xinference, i.e.
       ``http(s)://<xinference-endpoint>/api/oidc/callback``. The same URL
       must be registered as a valid redirect URI at the provider.

When ``XINFERENCE_OIDC_ENABLED`` is set, all four remaining variables are
required — Xinference refuses to start and reports the missing ones
otherwise.

Example (Keycloak):

.. code-block:: bash

   export XINFERENCE_OIDC_ENABLED=1
   export XINFERENCE_OIDC_ISSUER=https://keycloak.example.com/realms/myrealm
   export XINFERENCE_OIDC_CLIENT_ID=xinference
   export XINFERENCE_OIDC_CLIENT_SECRET=<client-secret>
   export XINFERENCE_OIDC_REDIRECT_URI=http://xinference.example.com:9997/api/oidc/callback
   xinference-local -H 0.0.0.0

Login flow
==========

* Direct users to ``GET /api/oidc/authorize`` on the Xinference endpoint.
  Xinference redirects the browser to the provider's authorization page
  (requesting the ``openid profile email`` scopes).
* After a successful provider login, the browser returns to
  ``/api/oidc/callback``. Xinference exchanges the authorization code,
  verifies the ID token signature against the provider's published JWKS,
  issues its own access and refresh tokens, and redirects to the web UI.

User provisioning and permissions
==================================

* Users are matched by the OIDC ``sub`` claim. On first login an account is
  created automatically with the provider's ``preferred_username`` (falling
  back to ``email``) as the username and ``source`` set to ``oidc``.
* Newly provisioned SSO users start with only the ``models:list``
  permission. An administrator grants further permissions through the
  **User Management** page or the ``/v1/admin/users`` API — changes take
  effect on the user's next request, without re-login.
* Disabling an SSO user in user management blocks their login even if their
  provider account is still active.
* OIDC users have no local password, so local password login and password
  change do not apply to them.

---

### Doc Source User Guide Pd Separation (`doc/source/user_guide/pd_separation.rst`)

.. _user_guide_pd_separation:

PD separation
=============

.. versionadded:: v3.4.0

Prefill/Decode (PD) separation is available in the community edition. Prefill
replicas compute prompt KV cache; decode replicas reuse it through Xavier or native NIXL and
produce the response. Requests are scheduled round-robin within each role.
No enterprise package or License is required.

Choose an engine and backend
----------------------------

For same-engine P/D, choose the engine first, then its Native NIXL or Xavier backend below. vLLM, SGLang and TRT-LLM support both; Xavier is the default. MLX supports Xavier only. Cross-engine deployments use Xavier and are described separately.

Native NIXL uses the engine's own P/D connector. Xavier's GPU transport also uses NIXL, with Xavier managing the handoff. The ``xavier_cache_backend`` setting (``native`` or ``lmcache``) selects storage for ordinary prefix-cache sharing, separately from P/D transport. LMCache support was added in v4.1.0. See :ref:`user_guide_vllm_enhancement` for storage configuration.

Launch
------

Python client
~~~~~~~~~~~~~

The example starts one prefill replica on GPU 0 and one decode replica on GPU 1.
Replace the worker address with the full ``ip:port`` reported by
``client.get_workers_info()``. Both replicas must use the same model and engine
configuration.

.. code-block:: python

   from xinference.client import Client

   client = Client("http://192.168.1.10:9997")
   client.launch_model(
       model_uid="qwen-pd",
       model_name="qwen2.5-instruct",
       model_size_in_billions="0_5",
       model_format="pytorch",
       quantization="none",
       model_engine="vLLM",
       replica=2,
       replica_config=[
           {
               "role": "prefill",
               "devices": [{"worker_ip": "192.168.1.10:9997",
                            "n_gpu": 1, "gpu_idx": [0]}],
           },
           {
               "role": "decode",
               "devices": [{"worker_ip": "192.168.1.10:9997",
                            "n_gpu": 1, "gpu_idx": [1]}],
           },
       ],
   )

The same ``replica_config`` is accepted by ``POST /v1/models``, the async Python
client, and the CLI ``--replica_config`` JSON option. In the Web UI, select vLLM, SGLang or MLX,
enable per-replica placement, and choose Prefill or Decode for each replica.
The community launch path defaults to Xavier when P/D roles appear.

CLI launch
~~~~~~~~~~

Start a local server first, or use an existing supervisor endpoint:

.. code-block:: bash

   xinference-local --host 127.0.0.1 --port 9997

In another terminal, launch a 1P+1D deployment:

.. code-block:: bash

   xinference launch --endpoint http://127.0.0.1:9997 \
     --model-uid qwen-pd --model-name qwen3 \
     --model-engine vLLM --size-in-billions 0_6 \
     --model-format pytorch --quantization none \
     --replica-config '[
       {"role":"prefill","devices":[{"worker_ip":"WORKER_ADDRESS","n_gpu":1,"gpu_idx":[0]}]},
       {"role":"decode","devices":[{"worker_ip":"WORKER_ADDRESS","n_gpu":1,"gpu_idx":[1]}]}
     ]'

Replace ``WORKER_ADDRESS`` with the worker's registered address, including its
actor port (shown in the worker startup log), not the HTTP API port. For workers
on different machines, use each worker's own address and GPU indices.
``--replica_config`` is an alias for ``--replica-config``. The replica count is
inferred from the array length; no separate Xavier switch is needed. Do not
combine per-replica placement with global ``--n-gpu``, ``--gpu-idx``, or
``--worker-ip``. Invalid PD roles or incomplete P/D topology fail before the
launch request is sent.

Configuration rules
~~~~~~~~~~~~~~~~~~~

* ``replica`` is the total number of P and D replicas. At least one of each role
  is required; more than one of either role is supported.
* Omitted roles default to ``hybrid`` (ordinary replicas). A PD deployment cannot
  mix hybrid replicas with P/D replicas.
* Each entry specifies one worker and its GPU allocation. Different replicas
  may run on different workers. Per-replica placement cannot be combined with
  global ``worker_ip``, ``gpu_idx`` or ``n_gpu``, or cross-worker sharding within
  a replica (``n_worker > 1``).
* ``vllm_transfer_backend_type`` (alias ``transfer_backend_type``) selects
  ``xavier`` (default) or ``nixl``. Unknown backend names are rejected.
* The topology is fixed for the deployment lifetime. To change its size or roles,
  terminate the model and relaunch it with the new configuration.

Inference and lifecycle
~~~~~~~~~~~~~~~~~~~~~~~

Send OpenAI-compatible chat/completion requests to the model UID used at launch. Both streaming and non-streaming responses are supported. Wait for all replicas to be ready before sending requests. Aborting a request stops work on both roles; terminating the model releases the deployment resources.

After a worker restart, relaunch the native NIXL PD deployment.

For fixed-length benchmarks, completion and chat requests accept ``ignore_eos=true``. Override model stop strings with an unused ``stop`` marker and verify the actual output token counts.

.. _user_guide_pd_same_engine:

Same-engine P/D
---------------

vLLM
~~~~

.. _native-nixl-backend:

.. _user_guide_pd_vllm_native:

Native NIXL
^^^^^^^^^^^

.. versionadded:: v4.0.0

Set ``vllm_transfer_backend_type="nixl"`` in the launch example to use vLLM's
native NixlConnector. Install ``nixl`` in the model environment together with
vLLM 0.21.0 or newer. The initial integration supports text-only models without
LoRA and requires TP=1, PP=1 and DP=1 per replica. Multiple P and D replicas are
supported. Xavier remains the default backend.

Native NIXL PD requires ``n=1`` per request; parallel sampling is not supported.

For NIXL, wildcard worker binds are supported through automatic host discovery. Set ``VLLM_NIXL_SIDE_CHANNEL_HOST`` in launch ``envs`` or the worker environment to select a reachable interface; launch ``envs`` take precedence. If discovery fails or returns loopback, configure this variable explicitly. Xinference always replaces ``VLLM_NIXL_SIDE_CHANNEL_PORT`` with a dynamically allocated port per replica, including on recovery. A fixed side-channel port is not supported; firewalls must allow dynamic ports between replica hosts.

.. _vllm-xavier-requirements-and-memory:

.. _user_guide_pd_vllm_xavier:

Xavier (default)
^^^^^^^^^^^^^^^^

.. versionchanged:: v4.0.0

   vLLM Xavier P/D uses GPU-to-GPU handoff by default. It requires Linux, NVIDIA GPUs, vLLM >= 0.21.0 and ``xoscar[nixl]>=0.11.1`` in both worker and model environments. Use a reachable host address (not ``0.0.0.0``). Missing NIXL fails the launch; there is no CPU fallback.

Xavier V1 requires one GPU per replica (TP=1, PP=1) and text-only models without LoRA. Multimodal models and prompt embeddings are not supported. See :ref:`xavier-cache-isolation` for request-scoped cache isolation.

Xavier's CPU cache can consume as much memory as the engine's GPU KV cache. Allow additional host memory for transfers when sizing your deployment.

Direct handoff and tiered history
"""""""""""""""""""""""""""""""""

.. versionadded:: v4.0.0

``xavier_gpu_cache_bytes`` defaults to ``268435456`` (256 MiB per replica) for P/D. History uses GPU first and spills to CPU when GPU capacity is exhausted. CPU history is bounded by the engine KV block capacity and can approach that cache size in host memory. CPU hits restore KV to GPU; misses compute locally. ``0`` disables history, retaining direct transfer. Transfer failures raise errors.

Direct handoff currently requires ``n=1``.

History is restored on P before the handoff to D. Long prompts may retain only a prefix; cache retention is best effort.

Unclaimed handoffs expire after 120 seconds. Claimed handoffs expire after 10 minutes without a scheduling retry or transfer progress. Active copies finish before release. Expired handoffs are recomputed locally on D.

The budget applies only to Xavier's GPU cache. Leave additional GPU memory for the model, vLLM's own KV cache and transfers. When this cache fills, Xavier stores reusable data in CPU memory; if no cache space is available, new data may not be retained for reuse.

Direct transfers and history loads can overlap with response generation for other requests.

Transfer failures and incompatible KV cache layouts raise errors instead of silently recomputing the request.

For implementation details, profiling and contributor tests, see the `Xavier developer README <https://github.com/xorbitsai/inference/blob/main/xinference/model/llm/vllm/xavier/README.md>`_.

Hybrid/recurrent attention limitation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. versionadded:: v4.0.0

Qwen3.5 text requests are supported with Xavier and native NIXL. Set ``language_model_only=True``, ``enable_prefix_caching=False``, ``mamba_cache_mode="none"``, ``disable_hybrid_kv_cache_manager=False`` and ``async_scheduling=False`` on both replicas. Use TP=1 and PP=1. Xavier history caching is not available for these hybrid/recurrent attention models; their configured history budget is not allocated. Image/video inputs and speculative decoding are not supported in this configuration. Native NIXL requires vLLM >= 0.22.0 for this mode.

SGLang
~~~~~~

.. _user_guide_pd_sglang_native:

Native NIXL
^^^^^^^^^^^

.. versionadded:: v4.0.0

Set ``model_engine="SGLang"`` and ``transfer_backend_type="nixl"`` with the launch example's P/D roles. This uses SGLang's NIXL backend through the same Xinference API and P/D router. Install ``nixl`` in each model environment. The initial native integration requires TP=PP=DP=1, one worker per replica, text requests, and no LoRA, speculative decoding or HiCache. Bootstrap ports are allocated per replica; both replica hosts must be reachable.

Native SGLang P/D requires one tokenizer worker; its internal HTTP endpoints use automatic per-replica authentication.

.. _sglang-with-xavier:

.. _user_guide_pd_sglang_xavier:

Xavier (default)
^^^^^^^^^^^^^^^^

.. versionadded:: v4.0.0

SGLang >= ``0.5.21`` accepts the same prefill/decode ``replica_config`` through the Python clients, REST API, CLI and Web UI. Change ``model_engine`` in the example to ``"SGLang"``. Explicit roles automatically enable Xavier; multiple prefill and decode replicas are supported.

SGLang Xavier P/D transfers KV directly between GPUs through Xavier's NIXL transport. Prefill and decode run concurrently using SGLang's native P/D lifecycle. Source and destination GPU slots remain owned until transfers complete; decode also receives the first token sampled by prefill. Streaming and non-streaming responses are supported. Transfer failures raise errors.

The model limitations are described in :ref:`user_guide_vllm_enhancement`. Install ``xoscar[nixl]>=0.11.1`` in worker and model environments and use reachable worker addresses. Missing NIXL fails launch without CPU fallback. Use the ``xavier`` transport; SGLang's own Mooncake backend is not exposed by this integration. For cross-engine P/D, use the per-replica engine settings below.

SGLang GPU P/D does not use ``xavier_cache_bytes`` or CPU HiCache. Retained Xavier GPU history is not yet supported; ``xavier_gpu_cache_bytes`` may be omitted or set to ``0``. Relaunch the deployment after a worker restart. Measure TTFT and throughput against ordinary replicas and native SGLang P/D for your workload.

Set ``XINFERENCE_SGLANG_XAVIER_TRANSFER_TIMEOUT`` in the worker environment or launch ``envs`` to change the Xavier handoff wait limit (default: 600 seconds). Increase it for long prefill queues or large prompts. Unfinished directory rooms expire after this interval; completed directory records expire after this interval unless the active decode request renews its lease. Both roles release them on completion.

SGLang Xavier P/D requires ``disaggregation_decode_enable_radix_cache=false``; each request transfers its full prompt KV. Prefill's local radix cache remains available. Enabling decode radix caching is rejected at launch.

TRT-LLM
~~~~~~~

.. versionadded:: v4.1.0

Use the same ``replica_config`` with ``model_engine="TRT-LLM"``, ``model_name="qwen3"``, ``model_size_in_billions="0_6"``, ``model_format="pytorch"`` and ``quantization="none"`` through the Python clients, REST API or CLI. Both roles require Linux/CUDA, TensorRT-LLM ``1.2.1``, BF16 dense Qwen3 weights and KV, one worker and one GPU per replica (TP=PP=CP=1). Chunked prefill and overlap scheduling must remain disabled. Native NIXL with different engines, quantization, multimodal input, LoRA, speculative decoding and ``cache_salt`` are unsupported.

Speculative decoding is supported on ordinary TRT-LLM replicas; it cannot be combined with P/D in this adapter. See :ref:`speculative_decoding` for launch options.

.. _user_guide_pd_trt_native:

Native NIXL
^^^^^^^^^^^

For native TRT P/D, set ``transfer_backend_type="nixl"`` and omit ``enable_xavier`` and ``xavier_cache_bytes``. TRT runs a context-only request on prefill, transfers its native context metadata and first sampled token, and resumes a generation-only request on decode using TRT's NIXL transceiver. NIXL must be available in both model environments. ``cache_transceiver_config.kv_transfer_timeout_ms`` defaults to 30000 and can be overridden in the launch arguments. The router releases the prefill request on completion, failure or cancellation. Measure both transports against ordinary replicas for your workload.

A ``cache_transceiver_config.kv_transfer_sender_future_timeout_ms`` value of ``None`` is normalized to 1000 ms so the executor can process cancellation even without a decode receiver. This does not change ``kv_transfer_timeout_ms=None``, which disables the overall transfer deadline.

.. _user_guide_pd_trt_xavier:

Xavier (default)
^^^^^^^^^^^^^^^^

The default Xavier path uses NIXL to transfer live 64-token KV pages, including a partially filled final page, directly between GPUs. A separate transfer process maps the TRT KV pool through CUDA IPC. Source slots stay pinned until GPU reads finish; decode waits for writes and replays the final prompt token. Install ``xoscar[nixl]>=0.11.1`` in each model environment and use reachable worker addresses. Omit ``xavier_cache_bytes`` and retained GPU history settings. Transfer failures raise errors without CPU staging or full-prefill fallback. Relaunch the model after a transfer-worker restart. Directory counters ``gpu_bytes`` and ``imported_tokens`` confirm completed imports; ``host_bytes`` stays zero.

Xavier loads KV asynchronously while other requests continue decoding. Cancelled decode requests retain their destination slots until all writes finish. Each transfer buffer is bounded by the maximum request KV size, with a 512 MiB target and a minimum of one KV page. Reserve GPU memory for both transfer buffers in addition to model weights and TRT's KV pool.

MLX
~~~

.. _mlx-with-xavier:

.. _user_guide_pd_mlx_xavier:

Xavier
^^^^^^

.. versionadded:: v4.0.0

MLX Xavier requires Apple silicon and ``mlx-lm>=0.31.2``. It supports unquantized, full-attention Qwen2, Qwen3 and Llama text models with one worker per replica. Weights and KV cache use FP16. Quantized weights or KV cache, rotating caches, hybrid attention, multimodal inputs, LoRA and speculative decoding are unsupported.

Use ``model_engine="MLX"``, ``model_format="mlx"`` and ``quantization="none"`` with the same prefill/decode roles. The example places one replica on each of two Mac workers. Replace the registered worker addresses with those reported by ``client.get_workers_info()``.

.. code-block:: python

   client.launch_model(
       model_uid="qwen-mlx-pd",
       model_name="qwen2.5-instruct",
       model_size_in_billions="0_5",
       model_engine="MLX",
       model_format="mlx",
       quantization="none",
       replica=2,
       xavier_cache_bytes=536870912,
       replica_config=[
           {"role": "prefill", "devices": [{"worker_ip": "MAC_P_WORKER:PORT",
                                          "n_gpu": "auto"}]},
           {"role": "decode", "devices": [{"worker_ip": "MAC_D_WORKER:PORT",
                                         "n_gpu": "auto"}]},
       ],
   )

MLX Xavier stores immutable CPU pages in a shared actor on the supervisor and transfers them through actor RPC. ``xavier_cache_bytes`` bounds this deployment cache (default: 512 MiB); model weights and per-replica Metal KV allocations consume additional memory. The prefill replica evaluates all but the last prompt token. Decode imports the prefix, evaluates that last token and samples the first response token. Streaming and non-streaming requests are supported. Active handoffs reserve cache capacity; insufficient capacity, incompatible metadata or transfer failure raises an error. The five-minute handoff deadline starts when prefill reserves capacity, before computing the prefix, and includes prefill and transfer time.

Each 64-token FP16 page uses ``2 * 64 * num_layers * num_kv_heads * head_dim * 2`` bytes (K and V, with two bytes per value). For example, 36 layers, 8 KV heads and a head dimension of 128 require 9 MiB per page; 512 MiB holds 56 pages, or 3584 prefix tokens. Round each prefix up to a whole page and size ``xavier_cache_bytes`` for the distinct pages reserved by all concurrent P/D requests. Ordinary shared-cache requests skip publication when their prefix exceeds the cache capacity or the 4096-page protocol limit.

Use the default ``xavier`` transport for MLX; native NIXL is unsupported. All workers must reach the supervisor's actor address. Relaunch after a worker restart. Two replicas on one Mac share its Metal GPU; measure your workload before expecting a throughput benefit.

.. _user_guide_pd_cross_engine:

Cross-engine P/D
----------------

.. _cross-engine-vllm-sglang-p-d:

Cross-engine vLLM/SGLang/TRT-LLM P/D
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Set ``model_engine`` and ``engine_config`` on each ``replica_config`` entry to select vLLM, SGLang or TRT-LLM independently for each role. Cross-engine P/D requires Xavier GPU transport. Engine options inherit launch arguments; ``engine_config`` overrides them per replica.

.. versionadded:: v4.0.0

   vLLM and SGLang in either prefill/decode direction.

.. code-block:: python

   client.launch_model(
       model_uid="qwen-cross-pd",
       model_name="qwen2.5-instruct",
       model_size_in_billions="0_5",
       model_format="pytorch",
       quantization="none",
       model_engine="vLLM",
       replica=2,
       replica_config=[
           {
               "role": "prefill", "model_engine": "vLLM",
               "engine_config": {"max_model_len": 8192},
               "devices": [{"worker_ip": "WORKER_ADDRESS",
                            "n_gpu": 1, "gpu_idx": [0]}],
           },
           {
               "role": "decode", "model_engine": "SGLang",
               "engine_config": {"context_length": 8192},
               "devices": [{"worker_ip": "WORKER_ADDRESS",
                            "n_gpu": 1, "gpu_idx": [1]}],
           },
       ],
   )

For vLLM/SGLang pairings, both workers require identical local weights, tokenizer assets and context limits. Supported models are unquantized FP16 full-attention Qwen2, Qwen3 and Llama text models with TP=PP=DP=1. Requires vLLM >=0.21.0, SGLang >=0.5.21 and ``xoscar[nixl]>=0.11.1``. Xavier checks contracts and prompt token IDs, converts 64-token KV pages on GPU and raises errors on incompatibility or transfer failure.

Streaming and non-streaming requests require ``n=1``. SGLang decode uses the first output token sampled by prefill. vLLM decode imports all but the final prompt token, computes that token and samples the output. Engine kernels can produce small numerical differences. vLLM decode requires at least two prompt tokens.

Cross-engine P/D does not support CPU fallback, retained Xavier history, logprobs, structured sampling, LoRA, speculative decoding, multimodal inputs or hybrid attention. Omit ``xavier_gpu_cache_bytes`` or set it to ``0``; do not set ``xavier_cache_bytes``. Relaunch after a worker restart. Native NIXL comparisons use the same engine on both roles.

.. _cross-engine-trt-llm-p-d:

.. versionadded:: v4.1.0

   TRT-LLM with vLLM or SGLang in either prefill/decode direction.

Xavier supports TRT-LLM prefill with vLLM or SGLang decode, and the reverse directions. All replicas use identical BF16 dense Qwen3 checkpoints, tokenizer assets and context limits, with one CUDA GPU per replica. TRT-LLM requires version ``1.2.1``; vLLM and SGLang requirements match the cross-engine section above. For CUDA/Metal deployments, see :ref:`NVIDIA and MLX cross-engine P/D ` below.

Set each replica's ``model_engine`` and ``engine_config`` independently. A topology containing TRT-LLM selects BF16 weights and KV for every replica; explicit FP16 options are rejected. Put engine-specific options in ``engine_config`` so they do not reach the other engine.

.. code-block:: python

   client.launch_model(
       model_uid="qwen3-trt-vllm-pd",
       model_name="qwen3",
       model_size_in_billions="0_6",
       model_engine="TRT-LLM",
       model_format="pytorch",
       quantization="none",
       model_path="/models/Qwen3-0.6B",
       replica=2,
       transfer_backend_type="xavier",
       replica_config=[
           {
               "role": "prefill",
               "model_engine": "TRT-LLM",
               "engine_config": {"max_seq_len": 4096},
               "devices": [{"worker_ip": "worker-a:9978", "gpu_idx": [0]}],
           },
           {
               "role": "decode",
               "model_engine": "vLLM",
               "engine_config": {"max_model_len": 4096},
               "devices": [{"worker_ip": "worker-b:9978", "gpu_idx": [0]}],
           },
       ],
   )

For SGLang, use ``model_engine="SGLang"`` and ``context_length`` instead of ``max_model_len``. Reverse the replica roles to reverse the transfer direction. Xavier converts TRT's head-major and the other engines' token-major KV layouts on GPU, preserving BF16 values, then transfers through NIXL. SGLang decode imports full prompt KV and the first sampled token; vLLM and TRT decode replay the final prompt token. Source and destination slots remain owned until GPU operations drain, including cancellation.

Cross-engine sampling restrictions above still apply. TRT additionally does not support ``cache_salt`` or tool calling through this adapter. Retained Xavier history and full-prefill fallback are disabled. CUDA-to-CUDA transfers use NIXL without CPU staging; ``gpu_bytes`` and ``imported_tokens`` measure completed imports and ``host_bytes`` stays zero.

NVIDIA and MLX cross-engine P/D
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. versionadded:: v4.0.0

   vLLM/SGLang with MLX in either prefill/decode direction.

Xavier supports both vLLM/SGLang prefill with MLX decode and MLX prefill with vLLM/SGLang decode, with streaming and non-streaming requests. FP16 64-token pages pass through CPU staging and actor RPC between CUDA and Metal. vLLM and MLX decode compute the last prompt token; SGLang decode imports the full prompt KV and the first output token sampled by P. MLX prefill publishes request-owned pages on its worker with a 512 MiB in-flight budget.

Use identical unquantized checkpoint and tokenizer files; do not convert the Mac checkpoint. Worker-local ``model_path`` and ``model_format`` belong in each replica's ``engine_config``. Xavier checks weights, tokenizer, effective model semantics, context limits and prompt token IDs. Model and sampling restrictions match vLLM/SGLang cross-engine P/D; failures raise errors.

.. code-block:: python

   client.launch_model(
       model_uid="qwen-nvidia-mac-pd",
       model_name="qwen2.5-instruct",
       model_size_in_billions="0_5",
       model_engine="vLLM",
       model_format="pytorch",
       quantization="none",
       replica=2,
       replica_config=[
           {
               "role": "prefill", "model_engine": "vLLM",
               "engine_config": {"model_path": "/models/Qwen2.5-0.5B-Instruct",
                                 "max_model_len": 8192},
               "devices": [{"worker_ip": "NVIDIA_WORKER:PORT",
                            "n_gpu": 1, "gpu_idx": [0]}],
           },
           {
               "role": "decode", "model_engine": "MLX",
               "engine_config": {"model_path": "/Users/me/models/Qwen2.5-0.5B-Instruct",
                                 "model_format": "mlx", "context_length": 8192},
               "devices": [{"worker_ip": "MAC_WORKER:PORT", "n_gpu": "auto"}],
           },
       ],
   )

To use MLX prefill, swap the two ``role`` values in the example. For SGLang on either role, use ``SGLang`` and replace ``max_model_len`` with ``context_length``. NVIDIA requires ``xoscar[nixl]>=0.11.1``; Mac requires ``xoscar>=0.11.1`` and ``mlx-lm>=0.31.2``. Workers must reach the supervisor and each other. Use Xavier transport, and omit ``xavier_gpu_cache_bytes`` or set it to ``0``. Relaunch after a worker restart.

For MLX prefill, ``xavier_cache_bytes`` sets the worker-local in-flight CPU page budget (default: 512 MiB). Capacity is reserved before Metal prefill, rounding the full prompt up to 64-token pages. A prompt exceeding the budget fails before prefill; concurrent requests wait for capacity. Reservations and transfer waits are bounded by ``XINFERENCE_SGLANG_XAVIER_TRANSFER_TIMEOUT`` (default: 600 seconds).

MLX-prefill cross-engine requests do not accept ``min_p``, ``presence_penalty``, ``frequency_penalty`` or ``seed``. The router rejects these parameters before scheduling either replica, so the first output token uses the same supported sampling settings as the rest of the response.

Run the supervisor and all worker/model environments with the same Python version and matching ``cloudpickle`` / ``tblib`` versions; actor RPC also carries exceptions and generators.

Compare end-to-end TTFT with the standalone decode engine on its own hardware, using the same effective FP16 weights and cache policy. Network transfers and CUDA/Metal copies may outweigh faster prefill. ``host_bytes`` measures CPU-staged payloads; ``imported_tokens`` records completed KV imports. Correct output alone does not prove reuse.

.. _trt-mlx-pd:
.. _trt-llm-and-mlx-over-a-lan:

.. versionadded:: v4.1.0

   TRT-LLM with MLX in either prefill/decode direction.

Xavier supports TRT-LLM prefill with MLX decode and MLX prefill with TRT-LLM decode across NVIDIA and Apple silicon workers. Use identical, unconverted BF16 dense Qwen3 checkpoints and tokenizer files on both hosts. Every replica uses BF16 weights and KV; FP16 overrides are rejected. The MLX replica automatically selects the unquantized BF16 registry entry. Both decode engines import the prefix and replay the last prompt token.

.. code-block:: python

   client.launch_model(
       model_uid="qwen3-trt-mlx-pd",
       model_name="qwen3",
       model_size_in_billions="0_6",
       model_engine="TRT-LLM",
       model_format="pytorch",
       quantization="none",
       download_hub="huggingface",
       replica=2,
       transfer_backend_type="xavier",
       replica_config=[
           {
               "role": "prefill", "model_engine": "TRT-LLM",
               "engine_config": {"model_path": "/models/Qwen3-0.6B",
                                 "max_seq_len": 4096},
               "devices": [{"worker_ip": "192.168.50.51:9978", "n_gpu": 1}],
           },
           {
               "role": "decode", "model_engine": "MLX",
               "engine_config": {"model_path": "/Users/me/models/Qwen3-0.6B",
                                 "model_format": "mlx", "context_length": 4096},
               "devices": [{"worker_ip": "192.168.50.120:9978", "n_gpu": "auto"}],
           },
       ],
   )

Swap the two roles for MLX prefill. Start workers with LAN-reachable addresses and ensure both workers and the supervisor can reach each other's actor ports. Specify the same ``download_hub`` explicitly. The Python and RPC dependency requirements and MLX-prefill memory budget above also apply. TRT-LLM requires ``1.2.1`` with one CUDA GPU; MLX requires Apple silicon and ``mlx-lm>=0.31.2``.

CUDA/Metal handoff uses bounded host-memory pages and actor RPC over the network. BF16 values travel as raw bytes without FP16 conversion. ``host_bytes`` and ``imported_tokens`` record completed imports; ``gpu_bytes`` stays zero for this transport. Streaming, cancellation and disconnect cleanup are supported. The TRT and MLX sampling restrictions above still apply. Measure TTFT against standalone decode on its own hardware, because network and device copies add latency.

---

### Doc Source User Guide Realtime (`doc/source/user_guide/realtime.rst`)

.. _realtime_sessions:

Realtime model sessions
=======================

Use ``WS /v1/models/{model_uid}/realtime`` for continuous media input. When authentication is enabled, send ``Authorization: Bearer <API_KEY>``; model permissions and the ``models:read`` scope apply. This is a Xinference protocol, not the OpenAI Realtime API.

The configured ``XINFERENCE_ALLOWED_IPS`` allowlist also applies before WebSocket upgrade.

The initial adapter is opt-in segmental transcription for PyTorch ``Fun-ASR-Nano-2512``. Launch with ``realtime=True`` and a FunASR version providing ``DynamicStreamingVAD``. It reuses loaded ASR and FSMN VAD weights; it does not load vLLM or promise incremental acoustic encoding. Other instances remain unchanged.

.. code-block:: python

   from xinference.client import Client

   client = Client("http://localhost:9997", api_key="<API_KEY>")
   uid = client.launch_model(
       model_name="Fun-ASR-Nano-2512",
       model_type="audio",
       model_engine="PyTorch",
       realtime=True,
   )

Send the configuration below, wait for ``session.started``, then send binary mono 16 kHz PCM16 little-endian audio without a WAV header. Send ``input.end`` to flush accepted audio, or ``session.cancel`` to cancel. Results use increasing ``seq`` values; replace cumulative ``sentences`` and mutable ``partial`` snapshots instead of appending them. Success requires one final output and ``session.closed`` with reason ``completed``.

.. code-block:: json

   {
     "event": "session.start",
     "protocol_version": 1,
     "task": "audio.transcription",
     "input": {
       "audio": {"encoding": "pcm_s16le", "sample_rate": 16000, "channels": 1}
     },
     "parameters": {"language": "zh", "hotwords": []}
   }

Only connection initialization uses Actor routing. Media and results use an authenticated persistent connection to the model process. Configure ``XINFERENCE_REALTIME_BIND_HOST`` and ``XINFERENCE_REALTIME_ADVERTISE_HOST`` when the Worker address is unsuitable. Keep internal ports on a trusted private network or encrypted tunnel; this endpoint does not provide TLS itself. Browser origins are rejected unless listed in ``XINFERENCE_REALTIME_ALLOWED_ORIGINS``; browser-native ticket authentication is not implemented.

Defaults are one realtime session per replica, 64 KiB input frames, 2 MiB output messages, eight-element input and output queues, 60 seconds of media inactivity, and 30 minutes per session. File requests and realtime sessions share request-limit accounting and inference locking. Cancellation retains capacity until an already running native operation finishes; safe unloading can therefore exceed the drain timeout. Real-GPU latency, accuracy and capacity still require deployment validation.

.. code-block:: bash

   export XINFERENCE_REALTIME_MAX_SESSIONS=1

Run the repository probe with a mono PCM16 WAV, using ``XINFERENCE_API_KEY`` from the environment. A 30–60 second speech recording must produce nonempty text before the last audio frame and ``input.end``. The probe cannot independently prove zero media RPC or resource cleanup; run the runtime integration tests as well.

.. code-block:: bash

   python benchmark/realtime_ws_probe.py \
     --url "ws://localhost:9997/v1/models/<MODEL_UID>/realtime" \
     --audio ./samples/zh_60s.wav \
     --chunk-ms 100 --require-interim \
     --report ./reports/nano_realtime.json

Nano launch options
-------------------

With ``realtime=True``, set ``realtime_options`` at model launch, not in session ``parameters``. In the Web UI, add an engine parameter named ``realtime_options`` with the JSON value below; the UI submits it as an object. REST/Python clients should send an object; JSON object strings are accepted for compatibility with older Web UIs. Unknown fields, non-object JSON and invalid value types are rejected.

.. code-block:: json

   {"optimize":true,"first_partial_ms":960,"partial_interval_ms":1500,"max_segment_ms":6000}

Optimization is disabled by default. This is a candidate configuration, not a latency guarantee. Removing ``realtime_options`` restores defaults; setting only ``optimize=false`` does not reset explicitly configured timing or segment limits.

---

### Doc Source User Guide Sd Webui (`doc/source/user_guide/sd_webui.rst`)

.. _sd_webui:

SD WebUI API compatibility
==========================

Xinference exposes Stable Diffusion WebUI-compatible routes alongside its
OpenAI-compatible image API. The extended workflow supports Diffusers Stable
Diffusion and SDXL models. Other image engines retain the basic generation
interface; they do not implement ControlNet, ADetailer, or Hires Fix.

Installation
------------

Install the optional dependencies on every worker that runs this workflow::

    pip install 'xinference[sdapi]'

Preprocessor-specific dependencies, such as MediaPipe, FaceXLib, and
InsightFace, are loaded only when their corresponding preprocessor is selected.
RealESRGAN and ADetailer download their weights on first use. Source licenses
and model-weight licenses are separate; vendored preprocessor notices and
licenses are included under ``xinference/thirdparty/controlnet``.

Generation
----------

Use the UID of an already launched image model in ``model`` or
``override_settings.sd_model_checkpoint``. For example::

    curl http://localhost:9997/sdapi/v1/txt2img \
      -H 'Content-Type: application/json' \
      -d '{"model":"sd-model","prompt":"a (small cat:1.2)",
           "seed":42,"steps":20,"width":512,"height":512,
           "batch_size":2,"n_iter":1,"request_id":"example-job"}'

The response contains base64 ``images``, the effective ``parameters``, and an
``info`` object with seeds and generation metadata. ``batch_size`` is the number
of images in each iteration; ``n_iter`` is the number of iterations. The seed
increments for each image, except when ``subseed_strength`` is nonzero. In that
case the primary seed stays fixed and the subseed increments.

Weighted prompts support parentheses, explicit weights, square brackets,
escaped delimiters, and long prompts split into CLIP token chunks. SDXL uses both
text encoders and supplies the pooled conditioning required by its pipeline.

For Hires Fix, set ``enable_hr`` to true, ``hr_scale`` (default 2),
``hr_upscaler`` (for example ``Latent``, ``Lanczos`` or ``R-ESRGAN 4x+``), and
optionally ``hr_second_pass_steps``. ``denoising_strength`` must be greater than
zero and at most one. The image is generated, enlarged, and refined in a second
image-to-image pass.

``POST /sdapi/v1/img2img`` accepts ``init_images`` containing raw base64,
image data URLs, or public HTTP(S) image URLs. It supports ``mask``,
``mask_blur``, ``inpainting_mask_invert``, ``inpaint_full_res``, and
``inpaint_full_res_padding``. ``resize_mode`` is 0 (resize), 1 (crop to fill), or
2 (fit and fill borders). URL downloads reject private addresses and validate
every redirect; image responses are limited to 64 MiB.

ControlNet and ADetailer
-----------------------

Pass WebUI scripts through ``alwayson_scripts``::

    {
      "ControlNet": {
        "args": [{"enabled": true, "module": "canny", "model": "controlnet-canny",
                  "image": "", "weight": 1.0,
                  "pixel_perfect": true, "resize_mode": "Crop and Resize"}]
      },
      "ADetailer": {
        "args": [true, {"ad_model": "face_yolov8n.pt", "ad_prompt": "[PROMPT]"}]
      }
    }

Select a ControlNet model listed by the launched model's family. Multiple units
are supported, with independent conditioning weights and guidance start/end.
The reference-only workflow is available for SD 1.5. Preprocessors include Canny,
depth, OpenPose, MLSD, and line art; individual detector dependencies and weights
must be available on the worker. IP-Adapter tensor-only workflows are not part
of this SDAPI port.

ADetailer runs after generation or Hires Fix and supports detection masks,
local inpainting, ``[PROMPT]``, ``[SEP]``, and ``[SKIP]`` prompt directives.

LoRA
----

Register a custom image model with ``model_family: "lora"`` and a local
``model_uri`` or a model hub repository containing one safetensors file. Reference
it as ``<lora:model-name:0.8>`` in the prompt. ``metadata.ss_output_name`` can supply
an alternative alias. ``override_settings.strict`` rejects unresolved LoRAs.
Adapters are removed after the request, including when inference fails.

An HTTPS CivitAI download URL can also be supplied as the registered LoRA's
``model_uri``. If authorization is needed, set ``CIVITAI_API_TOKEN`` in the worker
environment. Tokens are not forwarded to redirected CDN hosts. Downloads use a
temporary file followed by atomic replacement.

Querying and cancelling
-----------------------

The following routes use the same authentication requirements as image generation:

* ``GET /sdapi/v1/loras`` and ``GET /sdapi/v1/upscalers`` list available adapters
  and upscalers.
* ``GET /controlnet/model_list``, ``GET /controlnet/module_list``, and
  ``GET /controlnet/control_types`` query the running models and preprocessors.
* ``POST /controlnet/detect`` accepts ``controlnet_module``,
  ``controlnet_input_images`` (or the alias ``controlnet_images``),
  ``controlnet_processor_res``, and threshold parameters.
* ``GET /sdapi/v1/progress?request_id=example-job`` returns request progress.
* ``POST /sdapi/v1/interrupt`` accepts
  ``{"model":"sd-model","request_id":"example-job"}`` and cancels that request.
  This route requires an explicit model and request ID; it is not a global stop.

ComfyUI is outside this API compatibility layer.

---

### Doc Source User Guide Speculative Decoding (`doc/source/user_guide/speculative_decoding.rst`)

.. _speculative_decoding:

Speculative decoding
====================
Speculative decoding proposes several tokens and lets the target model verify
them together. The available methods and launch options depend on the engine.
Use a compatible drafter and measure latency and throughput for your workload.

Paired MTP drafter (Gemma 4)
----------------------------

Some models ship a small paired drafter checkpoint that predicts several tokens
ahead, which the target model then verifies in one pass. Output is unchanged,
decoding gets faster. Gemma 4 calls this multi-token prediction (MTP) and
publishes a ``*-it-assistant`` drafter for every variant.

Pass ``--enable_mtp true`` at launch to download the drafter declared by the
model spec and run it alongside the target model:

.. code-block:: bash

   xinference launch --model-name gemma-4 --model-engine vllm \
     --model-format pytorch --size-in-billions 12 --quantization none \
     --enable_mtp true

In the Web UI the same options live under *Advanced Configuration → Speculative
Decoding*, which only appears for a format/size that actually ships a drafter.

Optional parameters:

* ``--num_speculative_tokens <n>``: how many tokens the drafter proposes per
  round, including the bonus token. Left unset, MLX reads it from the drafter,
  which runs at the depth it was trained for (``4`` for Gemma 4). Gemma 4 on
  llama.cpp, vLLM, and SGLang follows the same model-size recipe: ``2`` for
  E2B, ``4`` for E4B and 26B-A4B, and the lower end (``4``) of the recommended
  ``4-8`` range for 12B and 31B. Other llama.cpp models keep xllamacpp's own
  default.
* ``--draft_quantization <quantization>``: which drafter conversion to use, when
  the spec declares more than one — the MLX build of Gemma 4 12B publishes
  eight. Defaults to the first declared, which is the least quantized one: a
  drafter is small and quantizing it costs acceptance rate, so pairing a
  quantized target with an unquantized drafter is the recommended setup.
* ``--draft_model_path ``: use a local drafter instead of the one declared
  by the spec. Implies ``--enable_mtp true``.

For these paired Gemma 4 drafters, the family and size must match the target —
the drafter shares the target's KV cache, so a mismatched checkpoint is rejected
rather than silently degrading.

Gemma 4 engine support:

.. list-table::
   :header-rows: 1
   :widths: 18 32 50

   * - Engine
     - Requirement
     - Notes
   * - :ref:`vLLM <vllm_backend>`
     - ``vllm>=0.22.0``, ``transformers>=5.8.0``
     - Translated into ``speculative_config`` with ``method: mtp``. An
       explicitly provided ``speculative_config`` is left untouched. Older
       vLLM treats the drafter as a generic draft model, while older
       Transformers does not recognize ``gemma4_assistant``; either case is
       rejected before engine initialization. Virtual-environment launches
       also synchronize ``flashinfer-cubin`` with ``flashinfer-python`` before
       starting vLLM, repairing stale environments that contain mismatched
       FlashInfer packages.
   * - :ref:`SGLang <sglang_backend>`
     - ``sglang==0.5.13.post1``, ``transformers==5.8.1``
     - Translated into ``--speculative-algorithm NEXTN`` with the matching
       ``speculative_num_steps`` / ``speculative_num_draft_tokens``. An
       explicitly provided ``speculative_algorithm`` is left untouched.
   * - :ref:`MLX <mlx_backend>`
     - ``mlx-vlm>=0.5.0`` (``>=0.6.1`` for Gemma 4 12B)
     - Served by the MLX vision engine, the one that runs multimodal models
       such as Gemma 4. The drafter is validated against the target when the
       model loads.
   * - :ref:`llama.cpp <llama_cpp_backend>`
     - ``xllamacpp>=2026.6.9713``
     - Translated into the ``draft-mtp`` speculative implementation. The
       drafter is a single gguf published inside the target's own repository,
       so its quantizations are its own (``BF16``, ``F16``, ``Q8_0`` for Gemma
       4) and independent of the target's. Earlier llama.cpp builds do not
       know the ``gemma4-assistant`` architecture and cannot load it.

Not supported by the Transformers engine: it runs its own continuous-batching
loop rather than ``generate()``, so there is nowhere to attach a drafter.

TRT-LLM
-------

.. versionadded:: v4.1.0

Ordinary TRT-LLM replicas support NGram and Draft/Target speculative decoding
with ``tensorrt-llm==1.2.1`` on Linux/CUDA. Model architectures and checkpoint
formats follow :ref:`trt_llm_backend`, with TP=PP=CP=1. Xavier shared caching, native P/D and Xavier
P/D do not support speculation through this adapter; see
:ref:`user_guide_pd_separation` for their supported configurations.

NGram reuses token patterns from the prompt and generated text without loading
a draft model. For a workload with repeated text, pass the native
``speculative_config`` at launch:

.. code-block:: python

   from xinference.client import Client

   client = Client("http://localhost:9997")
   common = dict(
       model_name="qwen3",
       model_size_in_billions="0_6",
       model_format="pytorch",
       model_engine="TRT-LLM",
       quantization="none",
       n_gpu=1,
       context_length=4096,
       enable_virtual_env=False,
   )
   client.launch_model(
       model_uid="qwen-ngram",
       **common,
       speculative_config={
           "decoding_type": "NGram",
           "max_draft_len": 3,
           "max_matching_ngram_size": 4,
           "is_public_pool": False,
       },
   )

Draft/Target loads a separate draft model. Provide its local checkpoint path
with ``draft_model_path``; ``num_speculative_tokens`` maps to TRT's
``max_draft_len`` and defaults to ``3``. This counts draft tokens and excludes
the target's bonus token. The draft and target must have compatible tokenizers
and vocabularies, and both must be supported by TRT-LLM 1.2.1. Use a smaller
dense Qwen3 checkpoint for the draft when serving a larger dense Qwen3 target.
This mode does not require the Gemma-style paired assistant or a shared KV cache.

.. code-block:: python

   client.launch_model(
       model_uid="qwen-draft-target",
       **dict(common, model_size_in_billions="1_7"),
       draft_model_path="/models/Qwen3-0.6B",
       num_speculative_tokens=3,
   )

The examples require a compatible TRT-LLM environment. Change the model sizes
and draft path to checkpoints available on your worker. A draft consumes
additional GPU memory and compute, so it may reduce throughput at high load.

An explicit ``speculative_config`` takes precedence over ``draft_model_path``
and ``num_speculative_tokens``. The adapter defaults
``disable_overlap_scheduler=True`` when speculation is configured. Use native
TRT options only with combinations supported by the pinned runtime. EAGLE3
and model-specific MTP require the corresponding architecture and speculative
checkpoint configuration. ``enable_mtp=True`` downloads the compatible drafter
declared in the Xinference model spec and configures Draft/Target unless
``speculative_config`` overrides it.

When it pays off
----------------
Speculation is only worth it when a drafting step is cheap *relative to* a
target decoding step. The drafter is a small dense model whose cost does not
change with the target, so the ratio is what decides the outcome — and a
mixture-of-experts target can land on the wrong side of it, because only its
activated slice is read per token:

.. list-table::
   :header-rows: 1
   :widths: 34 22 22 22

   * - gemma-4 MLX, 4bit, M5 Pro
     - Without a drafter
     - With MTP
     - Accepted per round
   * - 31B (dense, 18.4 GB read per token)
     - 14.9 tok/s
     - **31.0 tok/s** (2.1x)
     - 2.08 of 4
   * - 26B-A4B (MoE, ~2.2 GB read per token)
     - 73.2 tok/s
     - 65.9 tok/s (0.9x)
     - 1.40 of 4

The 0.83 GB drafter costs about 5% of a 31B decoding step but 39% of a
26B-A4B one, so on the MoE the three drafting steps of a round already exceed
one plain decoding step — and the round has to win that back from a lower
acceptance rate. The MoE is still the faster model here in absolute terms; it
simply has no headroom left for speculation.

So measure before leaving it on, and note that the verification step itself is
not the problem: a four-token forward costs only ~40% more than a single-token
one on either model.

---

### Doc Source User Guide Using Docker Swarm (`doc/source/user_guide/using_docker_swarm.rst`)

.. _using_docker_swarm:

Docker Swarm: Deployment Considerations
=======================================

This is a community experience report and architecture discussion, not a ready-to-run deployment guide. The contributor tested Xinference 2.2.0 with Docker 29.2.1, Ubuntu 22.04 and NVIDIA driver 570 on three machines, each with one NVIDIA L20. Xinference 3.x has not been validated in this Swarm setup.

Why consider Swarm?
-------------------

For a small team managing a dozen or so similar GPU servers, Swarm can reduce the manual work of deploying, restarting and updating containers across machines. It is built into Docker Engine. This was the contributor's reason for choosing it, not a rule that small clusters should always prefer Swarm.

Consider existing operational expertise, GPU management and availability requirements as well as cluster size. Teams already operating Kubernetes or needing its device-plugin ecosystem may prefer :ref:`using_kubernetes`.

Two layers of scheduling
------------------------

Swarm managers schedule container tasks on Docker nodes and maintain the services' desired state. Managers can also run tasks. A Xinference supervisor manages model placement and requests; Xinference worker processes register with it and run models. The supervisor does not provision worker containers: container orchestration and model serving are separate responsibilities.

A global Swarm service runs one task per eligible node, not one per GPU. GPU reservations and placement constraints must match the intended worker layout. Internal actor addresses must identify reachable individual processes; a load-balanced service address is not a substitute for each worker's address.

GPU management
--------------

Swarm generic resources are manually advertised scheduling resources, not automatic GPU discovery or health monitoring. With the resource kind ``NVIDIA-GPU``, Swarm exports allocated UUIDs through ``DOCKER_RESOURCE_NVIDIA-GPU``. NVIDIA runtime configuration must read that same variable: ``swarm-resource = "DOCKER_RESOURCE_NVIDIA-GPU"``. Scheduling reservations alone do not enforce GPU visibility.

Operators must keep advertised GPU UUIDs current and handle failed GPUs. This manual work was acceptable for the contributor's stable, homogeneous machines, but it is an important trade-off when hardware changes frequently. See the `NVIDIA runtime implementation <https://github.com/NVIDIA/nvidia-container-toolkit/blob/main/cmd/nvidia-container-runtime-hook/hook_config.go>`_ for how the configured variable is selected.

Persistence and recovery
------------------------

In current Xinference, disabling authentication does not make the supervisor stateless. Its default ``XINFERENCE_HOME`` contains launch history (including autostart configuration), monitoring configuration and system settings. Persist this state; persist authentication data and keys too when authentication is enabled. Environment variables for download settings do not replace these databases.

A host bind mount stays on that host. Use placement on the specific storage host, or suitable durable storage available after rescheduling. A manager-role constraint alone does not select a unique host in a multi-manager cluster. Persist worker caches and engine environments separately.

Swarm can replace failed containers, but that does not establish Xinference model recovery or uninterrupted service. Test worker reconnection and model recovery for the chosen release. A single-manager Swarm cannot reschedule tasks while that manager is unavailable; surviving tasks can continue running. See `Swarm administration <https://docs.docker.com/engine/swarm/admin_guide/>`_.

Further reading
---------------

See `Swarm concepts <https://docs.docker.com/engine/swarm/key-concepts/>`_ for the orchestration model, :ref:`using_docker_compose` for the existing Compose setup, and :doc:`/getting_started/migration_3_0` for Xinference 3.x changes. A complete Swarm configuration and current-version cluster validation are outside the scope of this report.

---

### Doc Source User Guide Vllm Enhancement (`doc/source/user_guide/vllm_enhancement.rst`)

.. _user_guide_xavier:
.. _user_guide_vllm_enhancement:

########################
Xavier: KV Cache Sharing
########################
Xavier enables KV cache sharing between model replicas, with adapters for
vLLM, SGLang, TRT-LLM and MLX. Replicas of the same engine can reuse compatible
prefix caches to reduce repeated prefill computation for long prompts and
multi-turn conversations.

Choose a deployment mode
------------------------

Use ordinary replicas to share compatible prefixes while each replica handles
complete requests. Shared prefix caching currently stays within the same
engine. For separate prefill and decode roles, including cross-engine and
CUDA/Metal deployments, use :ref:`user_guide_pd_separation`. P/D transport and
ordinary cache storage have separate configuration and compatibility limits.

Usage
-----

Launch multiple replicas with ``enable_xavier=True`` and select a supported ``model_engine``. See :ref:`user_guide_backends` for engine requirements and :ref:`user_guide_pd_separation` for P/D deployment.

The example launches two ordinary replicas. Choose the engine installed on your
workers; the model format and quantization must match it. Each CUDA replica
uses one GPU, while MLX runs on Apple silicon.

.. code-block:: python

   from xinference.client import Client

   client = Client("http://192.168.1.10:9997")
   engine = "vLLM"  # vLLM, SGLang, TRT-LLM or MLX
   model_format, quantization = {
       "vLLM": ("pytorch", "none"),
       "SGLang": ("pytorch", "none"),
       "TRT-LLM": ("pytorch", "none"),
       "MLX": ("mlx", "bf16"),
   }[engine]
   model_uid = client.launch_model(
       model_name="qwen3",
       model_size_in_billions="0_6",
       model_engine=engine,
       model_format=model_format,
       quantization=quantization,
       replica=2,
       n_gpu="auto" if engine == "MLX" else 1,
       context_length=4096,
       enable_xavier=True,
       enable_virtual_env=False,
   )

Due to the underlying communication not recognizing ``0.0.0.0``, the actual IP address needs to be passed when starting Xinference, for example: ``xinference-local -H 192.168.xx.xx``.

Engine requirements
-------------------

.. _xavier-vllm:
.. _limitations:

vLLM
~~~~

.. versionadded:: v1.2.0

V1 connector
^^^^^^^^^^^^

.. versionadded:: v3.4.0

The V1 connector requires vLLM >= ``0.21.0``, TP=1 and PP=1, with text-only
models and no LoRA. Multimodal and prompt-embedding inputs are unsupported.
Versions 0.11 through 0.20 are unsupported. Use the P/D guide for recurrent
model requirements and GPU history settings.

The vLLM GPU transfer path requires NVIDIA GPUs.

.. versionchanged:: v4.0.0

   For attention-only models on vLLM V1, Xavier preserves the model's ``enforce_eager`` setting, allowing CUDA Graph execution. Set ``enforce_eager=True`` at model launch to use eager execution. Recurrent models continue to use eager execution by default.

.. versionadded:: v4.0.0

   Retained vLLM GPU history, configured with ``xavier_gpu_cache_bytes``.

Legacy V0 adapter
^^^^^^^^^^^^^^^^^

The legacy adapter supports vLLM >= ``0.7.0`` and < ``0.11.0`` for ordinary
replica cache sharing. Use the V1 connector for current P/D deployments.

SGLang replicas
~~~~~~~~~~~~~~~

.. versionadded:: v4.0.0

SGLang >= ``0.5.21`` can share prefix KV cache through Xavier's HiCache storage backend. Launch multiple replicas with ``model_engine="SGLang"`` and ``enable_xavier=True``. The initial adapter supports unquantized FP16 Qwen2, Qwen3 and Llama text models with full attention and TP=PP=DP=1; LoRA, speculation, hybrid attention and multimodal inputs are unsupported.

Xavier loads BF16 checkpoints as FP16 and emits a warning. Both weights and KV caches use FP16.

``xavier_cache_bytes`` sets the deployment's shared CPU cache budget (default: ``536870912``, 512 MiB). Each SGLang replica also allocates its own GPU and HiCache host pools. Eviction or unavailable storage becomes a cache miss for ordinary replicas. Cache namespaces include the actual weights, tokenizer, model configuration, KV geometry and engine format; caches cannot yet be exchanged between vLLM and SGLang.

Ordinary SGLang replicas share CPU HiCache pages. Explicit prefill/decode roles use Xavier GPU-to-GPU transfer instead. For same-engine or cross-engine P/D and per-replica engine settings, see :ref:`user_guide_pd_separation`.

.. _xavier-trt-llm:

TRT-LLM replicas
~~~~~~~~~~~~~~~~

.. versionadded:: v4.1.0

TRT-LLM requires Linux/CUDA and ``tensorrt-llm==1.2.1``. Xavier supports
unquantized BF16 dense Qwen3 text models, one worker and one GPU per replica
(TP=PP=CP=1). Chunked prefill and overlap scheduling are disabled. Quantization,
hybrid attention, multimodal input, LoRA and speculative decoding are unsupported
with Xavier. See :ref:`trt_llm_backend` for the engine environment requirements.

Ordinary TRT replicas publish complete 64-token KV pages to a shared CPU cache
actor. ``xavier_cache_bytes`` sets the deployment budget (default: 512 MiB).
Partial pages and the final prompt token are computed locally; misses fall back
to local prefill. Model weights, engine KV pools and transfer buffers consume
additional memory. Xavier P/D instead transfers live KV pages through NIXL;
see :ref:`user_guide_pd_separation`.

MLX replicas
~~~~~~~~~~~~

.. versionadded:: v4.0.0

MLX Xavier requires Apple silicon and ``mlx-lm>=0.31.2``. It supports unquantized, full-attention Qwen2, Qwen3 and Llama text models with one worker per replica. Weights and KV cache use FP16. Quantized weights or KV cache, rotating caches, hybrid attention, multimodal inputs, LoRA and speculative decoding are unsupported.

``xavier_cache_bytes`` bounds the shared CPU cache (default: 512 MiB). Each replica also owns its model and Metal KV cache. Ordinary replicas compute locally on a miss or unavailable storage; incompatible deployments cannot share pages. Returned usage reports reused tokens in ``prompt_tokens_details.cached_tokens``. Explicit prefill/decode roles enable MLX Xavier P/D automatically. Cross-engine cache sharing is not yet supported.

Storage backends
----------------

Native storage
~~~~~~~~~~~~~~

The default ``xavier_cache_backend="native"`` uses each engine's Xavier cache
implementation. ``xavier_cache_bytes`` sets the shared CPU budget for SGLang,
TRT-LLM and MLX; it is not a vLLM setting. vLLM's GPU history uses
``xavier_gpu_cache_bytes``. These budgets exclude weights, engine KV pools and
transfer buffers. Choose the engine above before sizing storage.

LMCache
~~~~~~~

.. versionadded:: v4.1.0

LMCache storage for ordinary vLLM V1, SGLang and TRT-LLM replicas.

``xavier_cache_backend="native"`` is the default and keeps the existing CPU and GPU cache behavior. The optional ``lmcache`` backend adds external prefix-cache storage for ordinary vLLM V1, SGLang and TRT-LLM replicas. Install ``xinference[xavier-lmcache]`` in the Linux environment hosting storage: each vLLM transfer actor's environment, or the supervisor environment for SGLang and TRT-LLM. Model environments can remain separate. This adapter targets LMCache >= ``0.5.6`` and < ``0.6`` with the native extension for the selected L2 backend, and Python 3.10 through 3.13.

Select the backend when launching the model:

.. code-block:: python

   model_uid = client.launch_model(
       model_name="qwen3",
       model_engine="vLLM",
       model_format="pytorch",
       model_size_in_billions=8,
       quantization="none",
       replica=2,
       enable_xavier=True,
       xavier_cache_backend="lmcache",
       xavier_cache_backend_config={
           "cpu_bytes": 536870912,
           "disk_path": "/mnt/nvme/xavier",
           "disk_bytes": 10737418240,
           "read_timeout": 30.0,
       },
   )

``cpu_bytes`` bounds each LMCache instance's DRAM pool (default: 512 MiB). vLLM creates one instance per replica; SGLang and TRT-LLM share one supervisor-owned instance per deployment. ``disk_path`` must be absolute and local to that storage host, and ``disk_bytes`` is the positive disk capacity per instance. LMCache manages asynchronous writes, prefetch and eviction within these tiers. Disk eviction uses watermarks, so the configured disk capacity is not a filesystem quota. Xavier also uses bounded transient CPU staging and read copies, up to another ``2 * cpu_bytes`` per instance; engine KV pools, owned import buffers, serialization and transfer buffers consume additional memory.

``read_timeout`` bounds a cache lookup's wait in seconds (default: 30). Missing, unavailable or timed-out cache reads fall back to local prefill. At most one LMCache prefetch job runs per storage instance; a timed-out or cancelled job still releases its LMCache read locks. Reads exceeding the transient capacity return a bounded prefix or fall back to local prefill. Compare TTFT and read latency against prefill cost for your workload.

For filesystem storage, the LMCache backend requires multiple ordinary replicas of the same engine and preserves that engine's existing Xavier model, dtype and parallelism limits. P/D roles, legacy vLLM V0 and MLX are unsupported; P/D KV handoff remains a separate transport concern. Use backend ``cpu_bytes`` instead of the native ``xavier_cache_bytes`` or ``xavier_gpu_cache_bytes`` budgets. Backend selection does not enable cross-engine cache sharing or change existing cache-salt behavior. Each deployment and storage instance owns a separate namespace and directory; relaunching does not recover old entries. Graceful shutdown removes that directory; after a process crash, remove abandoned deployment directories when they are no longer in use.

Mooncake Store
^^^^^^^^^^^^^^

.. versionadded:: v4.2.0

Set ``l2_backend="mooncake"`` in ``xavier_cache_backend_config`` to use LMCache's Mooncake Store adapter. The default ``l2_backend="filesystem"`` preserves the disk configuration above. This selects storage for ordinary prefix-cache reuse; P/D transport and supported engine/model combinations remain the same.

The storage environment needs LMCache built with ``BUILD_MOONCAKE=1`` and compatible Mooncake SDK headers and libraries. Start the Mooncake master, metadata service and storage pool first. Use a reachable ``local_hostname`` without a port; Mooncake allocates a distinct port for each storage actor. Replace the endpoints in this TCP example with reachable service addresses, and pass the object as ``xavier_cache_backend_config``:

.. code-block:: json

   {
     "l2_backend": "mooncake",
     "cpu_bytes": 536870912,
     "storage_bytes": 10737418240,
     "read_timeout": 30,
     "mooncake_config": {
       "local_hostname": "storage-node",
       "metadata_server": "http://store-host:18080/metadata",
       "master_server_addr": "store-host:15051",
       "protocol": "tcp",
       "global_segment_size": 0,
       "local_buffer_size": 67108864
     }
   }

``storage_bytes`` bounds this actor's tracked remote cache capacity, together with ``cpu_bytes``; it is not a Mooncake cluster quota. At capacity, a new page is skipped and one least-recently-used owned object is scheduled for deletion. Only one deletion runs at a time, after outstanding LMCache writes drain; sustained traffic with new prefixes can therefore skip many new pages. Mooncake manages actual storage capacity and eviction; ``global_segment_size`` is the memory this client contributes. ``disk_path`` and ``disk_bytes`` apply only to filesystem storage and are rejected with Mooncake. ``protocol="rdma"`` additionally requires working RDMA hardware and drivers; LMCache registers the eagerly allocated L1 buffer.

Deployment, schema and cache-salt isolation still apply. vLLM uses a distinct storage namespace per replica rank; SGLang and TRT-LLM replicas share one supervisor-owned storage actor. Other deployments' objects are not discovered, and relaunching does not recover old entries. Cache misses, unavailable reads and timeouts fall back to local prefill. Graceful shutdown drains writes and deletes only this actor's tracked namespaced objects before closing its client. If cleanup fails or the process crashes, remaining objects are subject to Mooncake's eviction policy. Cleanup never explicitly deletes other clients' keys or stops the shared service. A client contributing memory withdraws that segment on close; use ``global_segment_size=0`` with a separate storage pool to avoid this.

.. _xavier-cache-isolation:

Request-scoped cache isolation
------------------------------

.. versionadded:: v4.1.0

Pass ``cache_salt`` with each inference request to isolate prefix-cache reuse. Requests with the same salt can reuse compatible caches across replicas; different salts cannot reuse each other's cached prefixes. This covers the local engine cache, Xavier shared caches and retained history, and same-engine or cross-engine P/D for vLLM, SGLang and MLX. Both P/D roles receive the same salt; incompatible cache scopes are rejected before importing KV.

With the Xinference Python client, include ``cache_salt`` in ``generate_config``. With the OpenAI client, use ``extra_body``:

.. code-block:: python

   client.chat.completions.create(
       model="my-model",
       messages=[{"role": "user", "content": "Hello"}],
       extra_body={"cache_salt": customer_cache_salt},
   )

The same top-level field is accepted by the Completions, Chat Completions, Responses and Anthropic Messages endpoints, for streaming and non-streaming inference. A salt must be a nonempty string of at most 256 characters without control characters. Unsupported model engines reject salted requests rather than ignoring the field. Legacy vLLM V0 requires Xavier to use salts.

Choose a stable, unguessable salt for each intended sharing boundary. A trusted application or gateway should authenticate callers, derive the salt from its own customer identity, and overwrite any client-supplied salt. Xinference does not map salts to users or create a tenant database. Salt knowledge permits cache sharing within that scope; client-selected salts alone do not enforce customer identity.

Without a salt, requests keep the existing shared-cache behavior. Set ``XINFERENCE_REQUIRE_CACHE_SALT=1`` in the API server environment to reject inference requests without a salt; the default is ``0``. Enable this policy after all callers or an API extension supply salts. The policy applies to the four inference endpoints above and does not disable prefix caching.

Distribution wrappers can supply an authoritative salt through the optional synchronous ``cache_salt(request)`` extension hook, returning ``xinference.extensions.CacheSalt(value)``. The hook runs after model-access checks and may read identity prepared by authenticated API dependencies from ``request.state``. Its result overrides the request field. Configure the extension through ``XINFERENCE_EXTENSIONS``; provider failures, missing return values and multiple providers reject the request without falling back to the client salt. When no provider is configured, the request field is used. Extension hooks must remain non-blocking.

Cache salting limits prefix-cache timing inference across sharing boundaries. It does not provide separate model processes, memory quotas or complete timing-side-channel isolation. Changing a salt starts a new cache scope; existing entries remain subject to normal eviction.

TRT-LLM does not support ``cache_salt`` through this adapter.

---

### Doc Source User Guide Zero Downtime Upgrade (`doc/source/user_guide/zero_downtime_upgrade.rst`)

.. _zero_downtime_upgrade:

Zero-downtime upgrades (Experimental)
=====================================

.. versionadded:: v4.1.0

Upgrade one component at a time while inference continues. Choose the component whose code or environment needs changing; replacing a control process keeps existing model processes and their loaded weights running.

Prepare an upgrade
------------------

The running deployment must already support the relevant replacement protocol and service host. API, Supervisor and Worker replacement support local and distributed deployments on Linux, macOS and Windows. Model-engine replacement has the Linux GPU requirements described below. On Windows, run original and replacement processes under the same user and privilege level. On Windows, the local service host and API subprocesses use selector event loops, each limited to 512 sockets.

By default, a replacement uses the component's current Python environment. No Python path is required. Use an administrator session when authentication is enabled.

In local mode, startup still uses an API parent and one shared Supervisor/Worker process. A component moves into a separate process only on its first upgrade; use the same commands and Web UI actions below. The original parent and shared process stay alive as service hosts, so stopping the original local service stops its replacement processes too. If a migrated Worker exits unexpectedly, the API and Supervisor stay available; attach a Worker to restore serving capacity.

.. note::

   An extension can optionally require controller sessions for Worker management RPCs. Callers must attach the versioned session and an operation ID; normal Supervisors do not add them automatically. A newer session is acknowledged only after management work drains, and failed drain keeps old sessions fenced. The provider must validate current authority on every check. Election and endpoint discovery are separate responsibilities. Worker replacement and model reload are rejected in this mode until session/resource transfer is supported. Without a provider, normal upgrade behavior is unchanged.

   Session metadata is also required through same-process ActorRefs. Startup recovery records are retained until an active session is installed. If installation happens during fencing or reconciliation, retry ``install_controller_session`` after activation to resume recovery. Recovery interrupted by a newer session preserves unfinished records.

   Worker-owned crash recovery and pending cache inventory publication acquire active-session admission for new events; already admitted operations keep their original session. Model OOM reports update only diagnostics for the current model incarnation and do not authorize management commands. Admitted termination finishes cleanup of its original model registration before drain completes. Interrupted native faults retain their intent and retry under active-session admission during installation or a later status report. This mechanism covers public management methods; private Python helpers remain internal. Lightweight heartbeats continue before session installation, during draining and after a drain timeout; they never register the Worker or resume recovery. Full status reports and their registration/recovery work still require active-session admission.

   A Supervisor authority provider can implement ``create_supervisor_control_authority(context)`` with a fixed process session, ``validate`` and ``close``. ``context.runtime`` supplies process-local ``pause(timeout)``, ``restore_worker(ref, replicas, incarnation)`` and ``resume()`` primitives. Optional async ``execute(operation, payload)`` and ``worker_registered(ref)`` hooks delegate extension-owned behavior. The runtime does not select Workers, probe models, confirm membership or orchestrate replacement. Old processes retain their original session. Fixed-session mode rejects native component replacement until session transfer is supported. These mechanisms add no failover command or HA service. Without a provider, normal upgrade behavior is unchanged. The port also provides a settings snapshot. Async provider callbacks execute on the actor loop and may overlap; providers synchronize their own services. A busy pause is rejected before management admission closes. Inference reads are unavailable before the first successful ``resume()``; afterwards, existing routes remain readable through later management pauses. Session-protected startup recovery retries outside the registration lock and retains persisted intent while access is fenced.

Check and install updates
-------------------------

.. versionadded:: v4.2.0

Use the version control at the bottom left of the Web UI to check for updates. Choose **Install and upgrade** to update Workers, then Supervisor, then API. Each component shows its progress; the displayed version changes after the API takes over. Detection uses the configured package source (``PIP_INDEX_URL``, or PyPI by default) on each component's machine and offers newer stable versions compatible with its Python.

.. figure:: /_static/upgrade_version_entry.png
   :alt: Open the update dialog from the version at the bottom left of the Web UI.
   :width: 360px
   :align: center

   Open the update dialog from the version at the bottom left of the Web UI.

.. figure:: /_static/upgrade_available.png
   :alt: Review available updates for each component, then choose Install and upgrade.
   :width: 100%
   :align: center

   Review available updates for each component, then choose Install and upgrade.

.. figure:: /_static/upgrade_installing.png
   :alt: Components upgrade one at a time. Expand Installation output to view the installation log.
   :width: 100%
   :align: center

   Components upgrade one at a time. Expand Installation output to view the installation log.

.. figure:: /_static/upgrade_completed.png
   :alt: When all components are updated, choose Refresh interface to load the new Web UI.
   :width: 100%
   :align: center

   When all components are updated, choose Refresh interface to load the new Web UI.

.. figure:: /_static/upgrade_up_to_date.png
   :alt: Check for updates again to confirm that all components report the new version.
   :width: 100%
   :align: center

   Check for updates again to confirm that all components report the new version.

Xinference installs the selected version and its dependencies into a separate directory under ``XINFERENCE_HOME/upgrades``. Status shows ``installing`` and ``migrating`` before the usual replacement states. Preparation failure leaves the current component serving. Existing model environments and service hosts keep running; prepared environments are retained for diagnosis and reuse.

Worker upgrades preserve the source environment's package versions, including inference engines and the Torch build. The new Worker environment reuses base Python packages and reinstalls the source environment's own packages. If the selected release requires incompatible dependencies, preparation fails and the existing Worker keeps serving; use a manually prepared compatible environment for that upgrade.

API and Supervisor upgrades also preserve source packages when guard protection or ``XINFERENCE_EXTENSIONS`` is enabled. Guard-enabled upgrades install the target release's ``guard`` extra. Dependency conflicts fail preparation while the current component keeps serving.

For preserved PyTorch builds with CUDA, CPU, ROCm or XPU version tags, Xinference first checks the configured package source. If that source cannot provide the build, uv selects the matching official PyTorch backend; other packages keep the configured source. The original build remains pinned. Bundled ``pip`` is excluded from the runtime package profile.

Use ``GET /v1/api/updates``, ``GET /v1/supervisor/updates`` or ``GET /v1/workers/{worker_address}/updates`` to check a component. Pass ``version`` to its existing relaunch endpoint to install and upgrade; ``version`` and ``python_executable`` are mutually exclusive. To use your own Xinference package source, set ``XINFERENCE_UPGRADE_INDEX_URL`` before starting the service on each relevant machine. The override applies to both detection and installation.

.. code-block:: python

    from xinference.client import Client

    client = Client("http://127.0.0.1:9997")
    update = client.check_component_update("api")
    if update["update_available"]:
        operation = client.relaunch_api(version=update["latest_version"])

A release can register optional ``xinference.migrations`` entry points in its package metadata. Each hook runs in the installed target environment before replacement and receives a dictionary with ``component``, ``source_version``, ``target_version`` and ``home``. Hooks must be safe to retry and compatible with components still serving. A hook failure stops the upgrade; the framework does not undo its data changes.

.. code-block:: toml

    [project.entry-points."xinference.migrations"]
    prepare = "xinference.deploy.migrations:migrate"

Use a custom package source
~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can publish your own Xinference builds to a PyPI-compatible package index, such as an internal repository or private mirror. The index must also provide or proxy the required dependencies. Keep the package name ``xinference`` and use a newer stable version compatible with the component's Python. Set ``XINFERENCE_UPGRADE_INDEX_URL`` to the index's simple API URL for both update checks and installation.

.. code-block:: bash

    export XINFERENCE_UPGRADE_INDEX_URL=https://packages.example.com/simple
    xinference-local --host 127.0.0.1 --port 9997

For a local package source, run the following helper from the checkout in the Xinference environment. It builds the current code into a Xinference wheel with the specified version and serves it over HTTP. Dependency requests are forwarded to ``PIP_INDEX_URL`` (or PyPI); use ``--upstream`` to select another dependency source.

.. code-block:: bash

    python scripts/mock_upgrade_index.py --version 9999.0.0 --port 18080

Run the package source on the same machine as the service, or use an address reachable by every component. For the local example above, set the URL below before starting Xinference. Use the version control at the bottom left to check for updates and install. After upgrading, confirm that ``GET /v1/cluster/version`` reports the version you published and the service still handles requests.

.. code-block:: bash

    export XINFERENCE_UPGRADE_INDEX_URL=http://127.0.0.1:18080/simple
    xinference-local --host 127.0.0.1 --port 9997

API and Web UI
--------------

In **Cluster Information**, choose **Restart / upgrade API**, or run:

.. code-block:: bash

    xinference relaunch-api --endpoint http://127.0.0.1:9997

.. figure:: /_static/supervisor_restart.png
   :alt: Restart or upgrade the API and Supervisor from Cluster Information.
   :width: 100%
   :align: center

   Restart or upgrade the API and Supervisor from Cluster Information.

The public HTTP address stays the same. New connections use the replacement API; existing requests and streams finish on the previous API. The Supervisor, Workers and models keep running. Refresh the browser to load the upgraded Web UI.

REST API: ``POST /v1/api/relaunch``.

The HTTP 202 response contains an ``operation_id``. Poll ``GET /v1/api-relaunches/{operation_id}`` or list operations with ``GET /v1/api-relaunches``. States are ``installing``, ``migrating``, ``starting``, ``switching``, ``draining``, ``succeeded`` and ``failed``. ``draining`` means the new API is serving while the previous API finishes admitted requests.

A failed startup leaves the previous API serving. Reuse ``request_id`` when retrying the same request. Only one replacement may run at a time. Operation history belongs to the stable service host and lasts until it stops. Supervisor upgrades are separate operations.

Supervisor
----------

In **Cluster Information**, choose **Restart / upgrade Supervisor**, or run:

.. code-block:: bash

    xinference relaunch-supervisor --endpoint http://127.0.0.1:9997

On Linux, macOS and Windows, ``xinference-supervisor`` and ``xinference-local`` can replace their Supervisor control process while retaining the API, Workers and model processes. The HTTP address and Supervisor actor address stay the same. New inference requests continue using the serving routes while the replacement prepares; existing streams stay on their model processes.

The replacement loads persisted configuration and checks the registered Workers' live model inventories and ownership. Management operations briefly wait during state transfer; model route lookups remain available. An active model launch, download, model reload or Worker handoff prevents replacement until that operation finishes. Failed preparation leaves the previous Supervisor serving.

REST API: ``POST /v1/supervisor/relaunch``.

The HTTP 202 response includes an ``operation_id``. Poll ``GET /v1/supervisor-relaunches/{operation_id}`` or list operations with ``GET /v1/supervisor-relaunches``. States are ``installing``, ``migrating``, ``starting``, ``preparing``, ``switching``, ``draining``, ``succeeded`` and ``failed``. Only one Supervisor replacement runs at a time. Reuse ``request_id`` when retrying the same operation; history lasts until the stable service host stops.

The service must have been started with Supervisor hosting support. The stable host and its state, cache, event, progress and model-routing actors retain their original environments. This operation upgrades Supervisor control logic; upgrading retained actors requires a planned restart.

.. _worker_hot_replacement:

Worker
------

Replace a Worker on Linux, macOS or Windows while retaining its model processes. The Supervisor and API server keep running. Both Workers and the model processes must support the handoff protocol and use Xoscar 0.12.2 or later, compatible Python versions, the same Supervisor, GPU visibility and extensions. Models spanning Workers and Xavier or P/D deployments are currently unsupported.

Worker relaunch requires the target Xinference version to support the running host's registration protocol. A downgrade to a version without that support fails while the current Worker keeps serving.

In **Cluster Information**, administrators can use **Restart / upgrade Worker** on a Worker. Operation records remain visible when its address changes or the page is refreshed. This retains the current model processes; model-engine upgrades use **Reconfigure / upgrade model** from **Running Models** or the model's detail view.

.. figure:: /_static/worker_restart.png
   :alt: Restart or upgrade a Worker from its row in Cluster Information.
   :width: 100%
   :align: center

   Restart or upgrade a Worker from its row in Cluster Information.

.. code-block:: python

    from xinference.client import Client

    client = Client("http://127.0.0.1:9997")
    operation = client.relaunch_worker("127.0.0.1:9978")
    status = client.get_worker_relaunch(operation["operation_id"])

To relaunch an existing Worker, send ``POST /v1/workers/{worker_address}/relaunch``. The Worker starts its replacement automatically and inherits its Supervisor, GPU visibility, runtime configuration and configured metrics exporter port; a new actor port is selected automatically.

The response is HTTP 202 with an ``operation_id``. Poll ``GET /v1/worker-relaunches/{operation_id}`` or list recent operations with ``GET /v1/worker-relaunches``. Status is one of ``installing``, ``migrating``, ``starting``, ``transferring``, ``pending``, ``succeeded`` or ``failed``. These routes require administrator access when authentication is enabled.

The Supervisor stores operation records in ``XINFERENCE_HOME/worker-relaunches.json``, so status remains available after the old Worker exits and can be reconciled after a Supervisor restart. Duplicate requests reuse an active operation. Keep the same optional ``request_id`` when retrying a request; use a new ID for a new operation. ``pending`` means the transfer outcome is uncertain: keep the processes alive and continue polling instead of starting another replacement.

The Supervisor also stores ownership reservations in ``XINFERENCE_HOME/worker-handoffs.json`` so an interrupted handoff can resume after a Supervisor restart. Replacement Workers keep updating the recovery file for the original Worker address, including models launched after relaunch.

Model management changes pause during handoff; inference calls continue. Failure before ownership transfer restores the old Worker. After transfer starts, keep both processes alive while the replacement retries. The old Worker stops automatically after successful replacement.

The model processes retain their original Xinference and engine code and loaded weights. This replaces the Worker control process. Deploy both processes on the same machine, and configure the service manager to let retained processes outlive the old Worker instead of killing its process group or container.

Service and container ownership
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Linux, macOS and Windows, ``xinference-worker`` automatically keeps a stable Worker host as its main process. Replacement Workers register before handoff, so retirement of the original Worker does not stop its models, replacement, service or container. Explicitly stopping the host terminates its owned processes. Custom launch commands can use the host wrapper below.

On Windows, stopping the Worker host forcibly terminates its owned processes. The Supervisor detects the stopped Worker after its heartbeat timeout.

.. code-block:: bash

    python -m xinference.deploy.worker_host xinference-worker \
      --endpoint http://127.0.0.1:9997 --host 0.0.0.0

A systemd service can keep its normal process-group cleanup, because the Worker host remains its main process.

For Docker, use the same host command as the container command. Install or mount prepared Python environments inside the running container. Keep that container running during upgrades: replacing or stopping the container terminates its model processes and cannot preserve their GPU allocations.

Model and engine
----------------

Open a running model's detail view and choose **Reconfigure / upgrade model**. Engine replacement requires **Keep GPU weights for reload** (``enable_weight_cache=true``) at launch, vLLM 0.31.0+ or SGLang 0.5.21+, Linux with CUDA or ROCm, one Worker, one replica and fixed tensor parallelism.

.. figure:: /_static/reconfigure_in_running.png
   :alt: Open model reconfiguration from Running Models.
   :width: 100%
   :align: center

   Open model reconfiguration from Running Models.

.. figure:: /_static/reconfigure_in_detail.png
   :alt: Open Reconfigure / upgrade model from the model's detail view.
   :width: 100%
   :align: center

   Open Reconfigure / upgrade model from the model's detail view.

For vLLM:

.. code-block:: python

    from xinference.client import Client

    client = Client("http://127.0.0.1:9997")
    operation = client.reload_model("my-model", {"max_num_seqs": 32})
    status = client.get_model_reload_status("my-model")

The current engine serves until its replacement is healthy, then existing requests finish on the old engine. Compatible engine and PyTorch versions can share retained weights, but each engine needs its own KV cache and CUDA graphs. Reserve extra GPU memory; incompatible versions also need a separate weight cache. Insufficient memory rejects the upgrade while the current engine keeps serving. See :ref:`launch_weight_cache_reload` for supported changes and limitations.

Send ``POST /v1/models/{model_uid}/reload`` to submit a model upgrade. Poll ``GET /v1/models/{model_uid}/reload`` until ``status`` is ``ready`` or ``error``. On error, ``restored=true`` confirms that the previous engine is serving.

Switch environments (optional)
------------------------------

You can also prepare a compatible Xinference environment manually. Pass its absolute Python path with ``python_executable`` in Python/REST calls or ``--python-executable`` in CLI commands. API/Supervisor paths are on the Supervisor machine; Worker/model paths are on the Worker machine. Omit ``version`` when using a prepared environment.

Upgrade boundaries
------------------

Keep service hosts and containers running throughout an upgrade. Stopping them also stops retained processes and releases GPU allocations. Supervisor support actors retain their original environments. Updating these hosts or retained support actors requires a planned restart; these operations do not upgrade an entire installation at once.

---

### Doc Source Models Builtin Index (`doc/source/models/builtin/index.rst`)

.. _models_builtin_index:

==============
Builtin Models
==============

.. toctree::
   :maxdepth: 1

   llm/index
   embedding/index
   image/index
   audio/index
   rerank/index
   video/index
   world/index

---

### Doc Source Models Model Abilities Audio (`doc/source/models/model_abilities/audio.rst`)

.. _audio:

=====
Audio
=====

Learn how to turn audio into text, synthesize speech, generate music, or extract
speaker embeddings with Xinference.


Introduction
==================


The Audio API provides four methods for interacting with audio:


* The transcriptions endpoint transcribes audio into the input language.
* The translations endpoint translates audio into English.
* The speech endpoint generates audio from the input text.
* The embeddings endpoint extracts a speaker embedding from an audio file.


.. list-table:: 
   :widths: 25  50
   :header-rows: 1

   * - API ENDPOINT
     - OpenAI-compatible ENDPOINT

   * - Transcription API
     - /v1/audio/transcriptions

   * - Translation API
     - /v1/audio/translations

   * - Speech API
     - /v1/audio/speech

   * - Speaker Embedding API
     - /v1/audio/embeddings


Invalid and silent audio
------------------------

* FunASR transcription models return HTTP 200 with an empty transcription when
  no speech is detected. An empty audio upload is invalid and returns HTTP 400.
* MegaTTS3 voice cloning requires decodable reference audio with samples and
  detectable sound. Empty, undecodable, or effectively silent ``prompt_speech``
  input, and empty ``prompt_latent`` input, return HTTP 400 before inference.


Supported models
-------------------

The audio API is supported with the following models in Xinference:

Audio to text
~~~~~~~~~~~~~

* :ref:`whisper-tiny <models_builtin_whisper-tiny>`
* :ref:`whisper-tiny.en <models_builtin_whisper-tiny.en>`
* :ref:`whisper-base <models_builtin_whisper-base>`
* :ref:`whisper-base.en <models_builtin_whisper-base.en>`
* :ref:`whisper-medium <models_builtin_whisper-medium>`
* :ref:`whisper-medium.en <models_builtin_whisper-medium.en>`
* :ref:`whisper-large-v3 <models_builtin_whisper-large-v3>`
* :ref:`whisper-large-v3-turbo <models_builtin_whisper-large-v3-turbo>`
* :ref:`Belle-whisper-large-v3-zh <models_builtin_belle-whisper-large-v3-zh>`
* :ref:`SenseVoiceSmall <models_builtin_sensevoicesmall>`
* :ref:`Paraformer-zh <models_builtin_paraformer-zh>`

Audio engines
~~~~~~~~~~~~~

Audio models with multiple implementations use one model name and select the
runtime with ``--model-engine``:

* The Whisper models listed above use ``transformers`` by default and also
  support ``MLX`` on Mac computers with Apple silicon.
* ``F5-TTS`` and ``Kokoro-82M`` use ``PyTorch`` by default and also support
  ``MLX`` on Mac computers with Apple silicon.
* ``SenseVoiceSmall`` and ``Fun-ASR-Nano-2512`` use ``PyTorch`` by default and
  also support ``MLX`` on Mac computers with Apple silicon.
* ``Qwen3-ASR-0.6B`` and ``Qwen3-ASR-1.7B`` use ``transformers`` by default. On
  Linux with NVIDIA GPUs, they can use ``vLLM`` for faster transcriptions; on
  Mac computers with Apple silicon, they can use ``MLX``.
* The Qwen3-TTS models, ``MeloTTS-English``, ``MeloTTS-English-v3``, and
  ``VoxCPM2`` use ``PyTorch`` by default and also support ``MLX`` on Mac
  computers with Apple silicon.

For example:

.. code-block:: bash

    xinference launch --model-name whisper-large-v3 --model-type audio --model-engine MLX
    xinference launch --model-name F5-TTS --model-type audio --model-engine MLX
    xinference launch --model-name Qwen3-ASR-1.7B --model-type audio --model-engine vLLM
    xinference launch --model-name Qwen3-TTS-12Hz-0.6B-Base --model-type audio --model-engine MLX

The former ``*-mlx`` model names remain accepted as launch compatibility
aliases. Registration, cache, version, and virtual-environment lookups use the
canonical model name. New integrations should use that name with
``--model-engine MLX``. The Web UI presents the available engines in the launch
dialog.


Text to audio (TTS)
~~~~~~~~~~~~~~~~~~~~~

**Models supporting zero-shot** (direct synthesis without reference audio):

* :ref:`ChatTTS <models_builtin_chattts>`
* :ref:`CosyVoice-300M-SFT <models_builtin_cosyvoice-300m-sft>`
* :ref:`CosyVoice-300M-Instruct <models_builtin_cosyvoice-300m-instruct>`
* MeloTTS series
* :ref:`Kokoro-82M <models_builtin_kokoro-82m>`
* :ref:`MegaTTS3 <models_builtin_megatts3>`

**Models supporting voice design** (natural-language voice description):

* :ref:`Breeze-TTS-2 <models_builtin_breeze-tts-2>`
* :ref:`FireRedTTS3-Instruct <models_builtin_fireredtts3-instruct>`
* :ref:`Qwen3-TTS-12Hz-1.7B-VoiceDesign <models_builtin_qwen3-tts-12hz-1.7b-voicedesign>`

Music generation
~~~~~~~~~~~~~~~~

* :ref:`ACE-Step1.5 <models_builtin_ace-step1.5>`
* :ref:`MiniMax-Music3 <models_builtin_minimax-music3>` (NVIDIA CUDA only)
* :ref:`YuE2-3B <models_builtin_yue2-3b>` (NVIDIA CUDA with BF16 support only)

Speaker embeddings
~~~~~~~~~~~~~~~~~~

* :ref:`speech_campplus_sv_zh-cn_16k-common <models_builtin_speech_campplus_sv_zh-cn_16k-common>`
* :ref:`speech_campplus_sv_zh_en_16k-common_advanced <models_builtin_speech_campplus_sv_zh_en_16k-common_advanced>`

**Models supporting voice cloning** (requires reference audio):

* :ref:`Breeze-TTS-2 <models_builtin_breeze-tts-2>`
* :ref:`CosyVoice-300M <models_builtin_cosyvoice-300m>`
* :ref:`CosyVoice 2.0 <models_builtin_cosyvoice2-0.5b>`
* :ref:`FishSpeech-1.5 <models_builtin_fishspeech-1.5>`
* :ref:`F5-TTS <models_builtin_f5-tts>`
* :ref:`FireRedTTS3-Base <models_builtin_fireredtts3-base>` and
  :ref:`FireRedTTS3-Instruct <models_builtin_fireredtts3-instruct>`
* :ref:`IndexTTS2 <models_builtin_indextts2>`
* :ref:`IndexTTS-2.5 <models_builtin_indextts-2.5>`

**Models supporting emotion control**:

* :ref:`IndexTTS2 <models_builtin_indextts2>`
* :ref:`IndexTTS-2.5 <models_builtin_indextts-2.5>`

Quickstart
===================

Speaker Embeddings
--------------------

The Speaker Embedding API accepts one audio file and returns one speaker
embedding. The built-in CAMPPlus models return a 192-dimensional vector. The
endpoint is intentionally stateless: applications can store the returned vectors
and use cosine similarity for speaker verification or 1:N speaker identification.
The request uses ``multipart/form-data``: ``model`` is the UID of a running
speaker-embedding model and ``file`` is the audio sample. Unlike the general
``/v1/embeddings`` endpoint, this endpoint returns one embedding object rather
than a list of text embeddings.

.. tabs::

  .. tab:: Web UI

    Open **Running Models**, select a running CAMPPlus model, and upload a clear
    speech sample in the **Speaker Embedding** panel. Select **Extract embedding**
    to inspect the vector and copy it from the results panel.

  .. code-tab:: bash cURL

    curl -X POST \
      'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/embeddings' \
      -H 'accept: application/json' \
      -F 'model=<MODEL_UID>' \
      -F '[email protected]'

  .. code-tab:: python Xinference Python Client

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
    model = client.get_model("<MODEL_UID>")

    with open("speaker.wav", "rb") as audio_file:
        result = model.create_embedding(audio_file.read())

    embedding = result["embedding"]

  .. code-tab:: json output

    {
      "object": "embedding",
      "model": "<MODEL_UID>",
      "dimensions": 192,
      "embedding": [0.0123, -0.0456, 0.0789]
    }

ModelScope decodes the input, converts multi-channel audio to one channel, and
resamples it to the model's 16 kHz sample rate. The returned vector preserves
the model output. Use cosine similarity when comparing two vectors; choose a
verification or identification threshold using representative audio from your
own application.

Transcription
--------------------

The Transcription API mimics OpenAI's `create transcriptions API <https://platform.openai.com/docs/api-reference/audio/createTranscription>`_.
We can try Transcription API out either via cURL, OpenAI Client, or Xinference's python client:

.. tabs::

  .. code-tab:: bash cURL

    curl -X 'POST' \
      'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/transcriptions' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "<MODEL_UID>",
        "file": "<audio bytes>",
      }'


  .. code-tab:: python OpenAI Python Client

    import openai

    client = openai.Client(
        api_key="cannot be empty", 
        base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1"
    )
    with open("speech.mp3", "rb") as audio_file:
        client.audio.transcriptions.create(
            model=<MODEL_UID>,
            file=audio_file,
        )

  .. code-tab:: python Xinference Python Client

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

    model = client.get_model("<MODEL_UID>")
    with open("speech.mp3", "rb") as audio_file:
        model.transcriptions(audio=audio_file.read())


  .. code-tab:: json output

    {
      "text": "Imagine the wildest idea that you've ever had, and you're curious about how it might scale to something that's a 100, a 1,000 times bigger. This is a place where you can get to do that."
    }



Translation
--------------------

The Translation API mimics OpenAI's `create translations API <https://platform.openai.com/docs/api-reference/audio/createTranslation>`_.
We can try Translation API out either via cURL, OpenAI Client, or Xinference's python client:

.. tabs::

  .. code-tab:: bash cURL

    curl -X 'POST' \
      'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/translations' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "<MODEL_UID>",
        "file": "<audio bytes>",
      }'


  .. code-tab:: python OpenAI Python Client

    import openai

    client = openai.Client(
        api_key="cannot be empty",
        base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1"
    )
    with open("speech.mp3", "rb") as audio_file:
        client.audio.translations.create(
            model=<MODEL_UID>,
            file=audio_file,
        )

  .. code-tab:: python Xinference Python Client

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

    model = client.get_model("<MODEL_UID>")
    with open("speech.mp3", "rb") as audio_file:
        model.translations(audio=audio_file.read())


  .. code-tab:: json output

    {
      "text": "Hello, my name is Wolfgang and I come from Germany. Where are you heading today?"
    }


Speech
--------------------

.. _audio_speech:

The Speech API mimics OpenAI's `create speech API <https://platform.openai.com/docs/api-reference/audio/createSpeech>`_.
We can try Speech API out either via cURL, OpenAI Client, or Xinference's python client:

Speech API use non-stream by default as

1. The stream output of ChatTTS is not as good as the non-stream output, please refer to: https://github.com/2noise/ChatTTS/pull/564
2. The stream requires ffmpeg<7: https://pytorch.org/audio/stable/installation.html#optional-dependencies

.. tabs::

  .. code-tab:: bash cURL

    curl -X 'POST' \
      'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/speech' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "<MODEL_UID>",
        "input": "",
        "voice": "echo",
        "stream": True,
      }'


  .. code-tab:: python OpenAI Python Client

    import openai

    client = openai.Client(
        api_key="cannot be empty",
        base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1"
    )
    client.audio.speech.create(
        model=<MODEL_UID>,
        input=,
        voice="echo",
    )

  .. code-tab:: python Xinference Python Client

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

    model = client.get_model("<MODEL_UID>")
    model.speech(
        input=,
        voice="echo",
        stream: True,
    )


  .. code-tab:: output

    The output will be an audio binary.


.. _breeze_tts_2_speech:

Breeze-TTS-2 Usage
~~~~~~~~~~~~~~~~~~

``Breeze-TTS-2`` supports English and Chinese voice design, voice cloning,
voice direction, and native streaming. It requires Linux and an NVIDIA CUDA
GPU. The model weights and self-hosted outputs are licensed for research and
non-commercial use only; review the upstream
`BreezeBlue model license <https://huggingface.co/BreezeBlue/Breeze-TTS-2/blob/main/LICENSE>`_
before launching the model.

For voice design, omit ``prompt_speech`` and pass a natural-language voice
description in ``instruct``. For voice cloning, pass reference audio bytes in
``prompt_speech`` and their exact transcript in ``prompt_text``. Voice direction
combines all three fields: ``prompt_speech``, ``prompt_text``, and ``instruct``.
``cfg_scale`` defaults to 1; the upstream project recommends 4 when stronger
instruction following is needed. ``seed`` defaults to 42. The model does not
support the OpenAI ``speed`` or preset ``voice`` controls.

``instruct``, ``prompt_text``, ``cfg_scale``, and ``seed`` use the existing
Speech API ``kwargs`` channel; raw REST requests encode ``kwargs`` as a JSON
string. ``prompt_speech`` is sent as a multipart file. The Xinference sync and
async clients handle both forms through their ``speech`` method.

.. code-block:: python

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
    model = client.get_model("<MODEL_UID>")

    # Voice design
    designed_voice = model.speech(
        input="Welcome aboard. Your journey begins now.",
        voice="",
        response_format="wav",
        instruct="A warm, thoughtful young woman with a calm delivery.",
        cfg_scale=4,
        seed=42,
    )

    # Voice cloning or voice direction
    with open("reference.wav", "rb") as reference_file:
        reference_audio = reference_file.read()
    directed_voice = model.speech(
        input="We need to discuss what happened last night.",
        voice="",
        response_format="wav",
        prompt_speech=reference_audio,
        prompt_text="This is the exact transcript of the reference audio.",
        instruct="Speak slowly with a restrained, serious tone.",
        cfg_scale=4,
        seed=42,
    )

Set ``stream=True`` to receive encoded audio chunks from the model's native
streaming runtime. CUDA Graph acceleration can be enabled at launch with
``--fast_all true`` or with the individual ``fast_text_encoder``,
``fast_backbone_prefill``, ``fast_backbone_decode``, ``fast_depth_decoder``,
and ``fast_codec`` model options. These modes increase startup work and GPU
memory use.


.. _ace_step1_5_speech:

ACE-Step1.5 Usage
~~~~~~~~~~~~~~~~~

The built-in registration uses the PyTorch engine and requires Python 3.11 or
3.12. It downloads the complete ACE-Step 1.5 checkpoint bundle from
`Hugging Face <https://huggingface.co/ACE-Step/Ace-Step1.5>`_ or
`ModelScope <https://modelscope.cn/models/ACE-Step/Ace-Step1.5>`_. The bundle
contains the default ``acestep-v15-turbo`` DiT, ``vae``,
``Qwen3-Embedding-0.6B``, and ``acestep-5Hz-lm-1.7B``. Xinference uses the
official `ACE-Step 1.5 Python API <https://github.com/ace-step/ACE-Step-1.5>`_
in a per-model virtual environment. ACE-Step is available under the
`MIT license <https://github.com/ace-step/ACE-Step-1.5/blob/main/LICENSE>`_.

This integration supports the bundled DiT, LM, and VAE checkpoints. Standalone
checkpoint combinations cannot currently be selected through ``config_path``
or ``lm_model_path``. ACE-Step supports CUDA, ROCm, Apple Silicon, Intel XPU,
and CPU; accelerator availability and performance depend on the system
PyTorch build.

``ACE-Step1.5`` reuses the Speech endpoint for text-to-music generation. Put
lyrics in ``input`` and the required music description in ``instruct`` inside
the existing ``kwargs`` field. Use ``[Instrumental]`` as the lyrics when
generating an instrumental track.

``duration`` defaults to 60 seconds. It accepts ``-1`` for model-selected
duration or a value from 10 through 600 seconds. ``seed=-1`` selects a random
seed; non-negative integers provide reproducible generation. Supported output
formats are ``aac``, ``flac``, ``mp3``, ``ogg``, ``opus``, ``wav``, and
``wav32``.
Generation is non-streaming, ``speed`` must be ``1.0``, and ``voice`` must be
``default``, an empty string, or null.

Launch the default DiT-only configuration, where ``thinking`` is false::

   xinference launch --model-name ACE-Step1.5 --model-type audio --model-engine PyTorch

To enable LM planning, metadata completion, and audio-code reasoning, load the
bundled 1.7B LM::

   xinference launch --model-name ACE-Step1.5 --model-type audio \
     --model-engine PyTorch --lm_model_path acestep-5Hz-lm-1.7B

This enables ``thinking=true`` and the related ``use_cot_*`` options. The LM
uses its PyTorch backend by default. ``offload_to_cpu``,
``offload_dit_to_cpu``, ``quantization``, and ``compile_model`` can be supplied
as launch options for supported hardware. ``lm_backend`` accepts ``pt``,
``vllm``, or ``mlx``; ``vllm`` requires CUDA for native execution, while
``mlx`` targets Apple Silicon. Other ACE-Step controls, including ``bpm``,
``keyscale``, ``timesignature``, ``inference_steps``, ``guidance_scale``,
``shift``, ``infer_method``, and ``timesteps``, can be supplied through the
same ``kwargs`` channel.

When the LM is loaded, ``thinking=false`` disables audio-code reasoning by
default. An explicitly enabled ``use_cot_caption``, ``use_cot_language``, or
``use_cot_metas`` still uses the LM for that planning step. MP3, AAC, and Opus
output require a working FFmpeg installation. OGG output is generated as WAV
and then encoded as OGG/Vorbis with libsndfile.

Raw REST requests encode ``kwargs`` as a JSON string. The Xinference sync and
async clients accept these names through their existing ``**kwargs`` argument.

.. tabs::

  .. code-tab:: bash cURL

    curl 'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/speech' \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "<MODEL_UID>",
        "input": "[Verse]\nMorning light across the city\n[Chorus]\nSing it back to me",
        "voice": "default",
        "response_format": "wav",
        "speed": 1.0,
        "stream": false,
        "kwargs": "{\"instruct\": \"Warm acoustic pop with intimate vocals\", \"seed\": 7, \"duration\": 60}"
      }' \
      --output ace-step.wav

  .. code-tab:: python Xinference Python Client

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
    model = client.get_model("<MODEL_UID>")
    wav = model.speech(
        input="[Verse]\nMorning light\n[Chorus]\nSing again",
        instruct="Warm acoustic pop with intimate vocals",
        voice="default",
        response_format="wav",
        seed=7,
        duration=60,
    )
    with open("ace-step.wav", "wb") as output:
        output.write(wav)


.. _minimax_music3_speech:

MiniMax-Music3 Usage
~~~~~~~~~~~~~~~~~~~~

``MiniMax-Music3`` reuses the Speech endpoint for text-to-music generation.
Put the lyrics in ``input`` and the required music description in
``instruct`` inside the existing ``kwargs`` field. Preserve line breaks and
put tags such as ``[Verse]`` and ``[Chorus]`` on their own lines.

``duration`` is the maximum generated length in seconds. Its range is 0.04
through 360 and its default is 60. Xinference passes it directly to the
Diffusers pipeline as ``audio_duration``. The model may emit an end-of-audio
token and finish before the limit.
Supported output formats are ``flac``, ``mp3``, ``ogg``, and ``wav``; WAV is
the default. Generation requires ``stream=false``, ``speed=1.0``, and
``voice`` set to ``default``, an empty string, or null.
Inference requires NVIDIA CUDA. Sampling steps and classifier-free guidance
remain at the Diffusers defaults and are not request parameters.

Xinference preserves the Diffusers pipeline's native 44.1 kHz stereo samples.
It wraps them in an IEEE-float WAV container without resampling or integer PCM
quantization. FLAC, MP3, and OGG responses are encoded from those samples
with libsndfile.

``instruct``, ``seed``, and ``duration`` are model options passed through the
existing ``kwargs`` channel rather than additional Speech API parameters. Raw
REST requests encode ``kwargs`` as a JSON string. The Xinference sync and async
clients accept these names through their existing ``**kwargs`` argument.

.. tabs::

  .. code-tab:: bash cURL

    curl 'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/speech' \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "<MODEL_UID>",
        "input": "[Verse]\nMorning light filtering through the pine\n[Chorus]\nSoftly the world begins to breathe",
        "voice": "default",
        "response_format": "wav",
        "speed": 1.0,
        "stream": false,
        "kwargs": "{\"instruct\": \"Warm acoustic pop with intimate female vocals, fingerpicked guitar, soft piano, and a wide final chorus.\", \"seed\": 7, \"duration\": 60}"
      }' \
      --output music3.wav

  .. code-tab:: python OpenAI Python Client

    import json

    from openai import OpenAI

    client = OpenAI(
        api_key="cannot be empty",
        base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1",
    )
    response = client.audio.speech.create(
        model="<MODEL_UID>",
        input="[Verse]\nMorning light\n[Chorus]\nSing again",
        voice="default",
        response_format="wav",
        extra_body={
            "kwargs": json.dumps({
                "instruct": "Warm acoustic pop with intimate vocals and soft piano.",
                "seed": 7,
                "duration": 60,
            })
        },
    )
    response.write_to_file("music3.wav")

  .. code-tab:: python Xinference Python Client

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
    model = client.get_model("<MODEL_UID>")
    wav = model.speech(
        input="[Verse]\nMorning light\n[Chorus]\nSing again",
        instruct="Warm acoustic pop with intimate vocals and soft piano.",
        voice="default",
        response_format="wav",
        seed=7,
        duration=60,
    )
    with open("music3.wav", "wb") as output:
        output.write(wav)


ChatTTS Usage
~~~~~~~~~~~~~

Basic usage, refer to :ref:`audio speech usage <audio_speech>`.

Fixed tone color. We can use fixed tone color provided by
https://github.com/6drf21e/ChatTTS_Speaker,
Download the `evaluation_result.csv <https://github.com/6drf21e/ChatTTS_Speaker/blob/main/evaluation_results.csv>`_ ,
take ``seed_2155`` as example, we get the ``emb_data`` of it.

.. code-block:: python

    import pandas as pd

    df = pd.read_csv("evaluation_results.csv")
    emb_data_2155 = df[df['seed_id'] == 'seed_2155'].iloc[0]["emb_data"]


Use the fixed tone color of ``seed_2155`` to generate speech.

.. code-block:: python

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

    model = client.get_model("<MODEL_UID>")
    resp_bytes = model.speech(
        voice=emb_data_2155,
        input=<The text to generate audio for>
    )


CosyVoice Usage
~~~~~~~~~~~~~~~

CosyVoice has two versions: CosyVoice 1.0 and CosyVoice 2.0. CosyVoice 1.0 has three different models:

- **CosyVoice-300M-SFT**: Choose this model if you just want to convert text to audio. There are pretrained voices available: ['中文女', '中文男', '日语男', '粤语女', '英文女', '英文男', '韩语女']
- **CosyVoice-300M**: Choose this model if you want to clone voice or convert text to audio in different languages. The ``prompt_speech`` is always required and should be a WAV file. For optimal performance, use a sample rate of 16,000 Hz.
- **CosyVoice-300M-Instruct**: Choose this model If you need precise control over the tone and pitch.

Basic usage, launch model ``CosyVoice-300M-SFT``.

.. tabs::

  .. code-tab:: bash cURL

    curl -X 'POST' \
      'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/audio/speech' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "<MODEL_UID>",
        "input": "",
        # ['中文女', '中文男', '日语男', '粤语女', '英文女', '英文男', '韩语女']
        "voice": "中文女"
      }'

  .. code-tab:: python OpenAI Python Client

    import openai

    client = openai.Client(
        api_key="cannot be empty",
        base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1"
    )
    response = client.audio.speech.create(
        model=<MODEL_UID>,
        input=,
        # ['中文女', '中文男', '日语男', '粤语女', '英文女', '英文男', '韩语女']
        voice="中文女",
    )
    response.stream_to_file('1.mp3')

  .. code-tab:: python Xinference Python Client

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

    model = client.get_model("<MODEL_UID>")
    speech_bytes = model.speech(
        input=,
        # ['中文女', '中文男', '日语男', '粤语女', '英文女', '英文男', '韩语女']
        voice="中文女"
    )
    with open('1.mp3', 'wb') as f:
        f.write(speech_bytes)


Clone voice, launch model ``CosyVoice-300M``.

.. code-block::

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

    model = client.get_model("<MODEL_UID>")

    zero_shot_prompt_text = ("")
    # The words said in the audio file should be identical
    # to zero_shot_prompt_text.
    #
    # The audio input file must be in WAV format.
    # For optimal performance, use a 16,000 Hz sample rate.
    #
    # Files with different sample rates will be resampled to 16,000 Hz,
    # which may increase processing time.
    with open(zero_shot_prompt_file, "rb") as f:
        zero_shot_prompt = f.read()

    speech_bytes = model.speech(
        "",
        prompt_text=zero_shot_prompt_text,
        prompt_speech=zero_shot_prompt,
    )


Cross lingual usage, launch model ``CosyVoice-300M``.

.. code-block::

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

    model = client.get_model("<MODEL_UID>")

    # The audio input file must be in WAV format.
    # For optimal performance, use a 16,000 Hz sample rate.
    #
    # Files with different sample rates will be resampled to 16,000 Hz,
    # which may increase processing time.
    with open(cross_lingual_prompt_file, "rb") as f:
        cross_lingual_prompt = f.read()

    speech_bytes = model.speech(
        "",  # text could be another language
        prompt_speech=cross_lingual_prompt,
    )

Instruction based, launch model ``CosyVoice-300M-Instruct``.

.. code-block::

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

    model = client.get_model("<MODEL_UID>")

    response = model.speech(
        "在面对挑战时，他展现了非凡的**勇气**与**智慧**。",
        voice="中文男",
        instruct_text="Theo 'Crimson', is a fiery, passionate rebel leader. "
        "Fights with fervor for justice, but struggles with impulsiveness.",
    )

CosyVoice 2.0 only has one model, it provides all the capabilities of the three CosyVoice models. The usage is the same as CosyVoice.

CosyVoice 2.0 stream usage, launch model ``CosyVoice2-0.5B``.

.. code-block::

    # Launch model
    from xinference.client import Client

    model_uid = client.launch_model(
        model_name=model_name,
        model_type="audio",
        download_hub="modelscope",
    )

    endpoint = "http://127.0.0.1:9997"
    input_string = "你好，我是通义生成式语音大模型，请问有什么可以帮您的吗？"

    # Stream request by openai client
    import openai
    import tempfile

    openai_client = openai.Client(api_key="not empty", base_url=f"{endpoint}/v1")
    # ['中文女', '中文男', '日语男', '粤语女', '英文女', '英文男', '韩语女']
    response = openai_client.audio.speech.with_streaming_response.create(
        model=model_uid, input=input_string, voice="英文女"
    )
    with tempfile.NamedTemporaryFile(suffix=".mp3", delete=True) as f:
        response.stream_to_file(f.name)
        assert os.stat(f.name).st_size > 0

    # Stream request by xinference client
    response = model.speech(input_string, stream=True)
    assert inspect.isgenerator(response)
    with tempfile.NamedTemporaryFile(suffix=".mp3", delete=True) as f:
        for chunk in response:
            f.write(chunk)


More instructions and examples, could be found at https://fun-audio-llm.github.io/ .


FishSpeech Usage
~~~~~~~~~~~~~~~~

Basic usage, refer to :ref:`audio speech usage <audio_speech>`.

Clone voice, launch model ``FishSpeech-1.5``. Please use `prompt_speech` instead of `reference_audio`
and `prompt_text` instead of `reference_text` to clone voice from the reference audio for the FishSpeech model.
This arguments is aligned to voice cloning of CosyVoice.

.. code-block::

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

    model = client.get_model("<MODEL_UID>")

    # The reference audio file is the voice file
    # the words said in the file should be identical to reference_text
    with open(reference_audio_file, "rb") as f:
        reference_audio = f.read()
    reference_text = ""  # text in the audio

    speech_bytes = model.speech(
        "<The text to generate audio for>",
        prompt_speech=reference_audio,
        prompt_text=reference_text
    )



Paraformer Usage
~~~~~~~~~~~~~~~~

+-------------------------------------------------------+-----+------+------------+---------+---------+
| model                                                 | vad | punc | timestamp  | speaker | hotword |
+=======================================================+=====+======+============+=========+=========+
| :ref:`models_builtin_paraformer-zh`                   | yes | yes  | no         | no      | no      |
+-------------------------------------------------------+-----+------+------------+---------+---------+
| :ref:`models_builtin_paraformer-zh-hotword`           | yes | yes  | no         | no      | yes     |
+-------------------------------------------------------+-----+------+------------+---------+---------+
| :ref:`models_builtin_paraformer-zh-spk`               | yes | yes  | yes        | yes     | no      |
+-------------------------------------------------------+-----+------+------------+---------+---------+
| :ref:`models_builtin_paraformer-zh-long`              | yes | yes  | yes        | yes     | no      |
+-------------------------------------------------------+-----+------+------------+---------+---------+
| :ref:`models_builtin_seaco-paraformer-zh` (recommend) | yes | yes  | yes        | yes     | yes     |
+-------------------------------------------------------+-----+------+------------+---------+---------+

1. **VAD & Punctuation Usage**

   All Paraformer models support VAD and punctuation.

2. **Timestamp & Speaker Usage**

   Only the following models support `timestamp` and `speaker`:
   
   - `paraformer-zh-spk`
   - `paraformer-zh-long`
   - `seaco-paraformer-zh`

   Among them, only `paraformer-zh-spk` enables **speaker info by default**.

   If you need speaker info when using `paraformer-zh-long` or `seaco-paraformer-zh`:

   - In Web UI: add an extra parameter with key ``spk_model`` and value ``cam++``
   - In command line: add the option ``--spk_model cam++``

   Example:

   .. code-block:: python

      from xinference.client import Client
      client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
      model = client.get_model("seaco-paraformer-zh")
      with open("asr_example.wav", "rb") as audio_file:
          audio = audio_file.read()
          model.transcriptions(audio, response_format="verbose_json")

3. **Hotword Usage**

   Only the following models support `hotword`:
   
   - `paraformer-zh-hotword`
   - `seaco-paraformer-zh`

   Example:

   .. code-block:: python

      from xinference.client import Client
      client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
      model = client.get_model("seaco-paraformer-zh")
      with open("asr_example.wav", "rb") as audio_file:
          audio = audio_file.read()
          model.transcriptions(audio, hotword="小艾 魔搭")



SenseVoiceSmall Offline Usage
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Now SenseVoiceSmall use a small vad model ``fsmn-vad``, it will be downloaded thus network required.

For offline environment, you can download the vad model in advance.

Download from `huggingface <https://huggingface.co/funasr/fsmn-vad>`_ or `modelscope <https://modelscope.cn/models/iic/speech_fsmn_vad_zh-cn-16k-common-pytorch/files>`_.
Assume downloaded to ``/path/to/fsmn-vad``.

Then when launching SenseVoiceSmall with Web UI, you can add an additional parameter with key ``vad_model`` and value ``/path/to/fsmn-vad`` which is the downloaded path.
When launching with command line, you can add an option ``--vad_model /path/to/fsmn-vad``.


Kokoro Usage
~~~~~~~~~~~~

The Kokoro model supports multiple languages, but the default language is English.
If you want to use other languages, such as Chinese, you need to install additional dependency packages
and add an additional parameter when starting the model.

1. pip install misaki[zh]

2. Initialize the model with the parameter lang_code='z',
   For all available ``lang_code`` options,
   please refer to `kokoro source code <https://github.com/hexgrad/kokoro/blob/main/kokoro/pipeline.py#L22>`_.
   If the model is started through the web UI, an additional
   parameter needs to be added, with the key as ``lang_code`` and the value as ``z``.
   If the model is started through the xinference client, the parameters are passed via the launch_model interface:

   .. code-block::

       model_uid = client.launch_model(
           model_name="Kokoro-82M",
           model_type="audio",
           compile=False,
           download_hub="huggingface",
           lang_code="z",
       )

3. When inferring, the voice must start with 'z', for example: ``zf_xiaoyi``.
   The currently supported voices are: https://huggingface.co/hexgrad/Kokoro-82M/tree/main/voices. For example:

   .. code-block::

       input_string = "重新启动即可更新"
       response = model.speech(input_string, voice="zf_xiaoyi")


IndexTTS2 Usage
~~~~~~~~~~~~

The IndexTTS2 model supports emotion control, you can use this feature by specifying some additional parameters.
Here are several examples of how to use IndexTTS2:

1. Synthesize new speech with a single reference audio file (voice cloning):

   .. code-block::

        from xinference.client import Client
        client = Client("http://0.0.0.0:6735")
        model = client.get_model("IndexTTS2")

        with open("../mp3_test_voice.mp3", "rb") as f:
            test_prompt_speech = f.read()

        response = model.speech(
            input="Translate for me, what is a surprise!",
            prompt_speech=test_prompt_speech,
        )

2. Using a separate, emotional reference audio file to condition the speech synthesis:

   .. code-block::

        from xinference.client import Client
        client = Client("http://0.0.0.0:6735")
        model = client.get_model("IndexTTS2")

        with open("../mp3_test_voice.mp3", "rb") as f:
            test_prompt_speech = f.read()

        with open("example/emo_sad.wav", "rb") as f:
            emo_prompt_speech = f.read()

        response = model.speech(
            input="It's such a shame the singer didn't make it to the finals.",
            prompt_speech=test_prompt_speech,
            emo_audio_prompt=emo_prompt_speech
        )

3. When an emotional reference audio file is specified, you can optionally set
   the ``emo_alpha`` to adjust how much it affects the output.
   Valid range is ``0.0 - 1.0`` , and the default value is ``1.0`` (100%):

   .. code-block::

        from xinference.client import Client
        client = Client("http://0.0.0.0:6735")
        model = client.get_model("IndexTTS2")

        with open("../mp3_test_voice.mp3", "rb") as f:
            test_prompt_speech = f.read()

        with open("example/emo_sad.wav", "rb") as f:
            emo_prompt_speech = f.read()

        response = model.speech(
            input="It's such a shame the singer didn't make it to the finals.",
            prompt_speech=test_prompt_speech,
            emo_audio_prompt=emo_prompt_speech,
            emo_alpha=0.9
        )

4. It's also possible to omit the emotional reference audio and instead provide
   an 8-float list specifying the intensity of each emotion, in the following order:
   ``[happy, angry, sad, afraid, disgusted, melancholic, surprised, calm]`` .
   You can additionally use the ``use_random`` parameter to introduce stochasticity
   during inference; the default is ``False`` , and setting it to ``True`` enables
   randomness:

   .. code-block::

        from xinference.client import Client
        client = Client("http://0.0.0.0:6735")
        model = client.get_model("IndexTTS2")

        with open("../mp3_test_voice.mp3", "rb") as f:
            test_prompt_speech = f.read()

        response = model.speech(
            input="Wow, I'm so lucky!",
            prompt_speech=test_prompt_speech,
            emo_vector=[0, 0, 0, 0, 0, 0, 0.45, 0],
            use_random=False
        )

5. Alternatively, you can enable ``use_emo_text`` to guide the emotions based on
   your provided ``text`` script. Your text script will then automatically
   be converted into emotion vectors.
   It's recommended to use ``emo_alpha`` around 0.6 (or lower) when using the text
   emotion modes, for more natural sounding speech.
   You can introduce randomness with ``use_random`` (default: ``False``;
   ``True`` enables randomness):

   .. code-block::

        from xinference.client import Client
        client = Client("http://0.0.0.0:6735")
        model = client.get_model("IndexTTS2")

        with open("../mp3_test_voice.mp3", "rb") as f:
            test_prompt_speech = f.read()

        response = model.speech(
            input="Quick, hide! He's coming! He's coming to get us!",
            prompt_speech=test_prompt_speech,
            emo_alpha=0.6,
            use_emo_text=True,
            use_random=False
        )

6. It's also possible to directly provide a specific text emotion description
   via the ``emo_text`` parameter. Your emotion text will then automatically be
   converted into emotion vectors. This gives you separate control of the text
   script and the text emotion description:

   .. code-block::

        from xinference.client import Client
        client = Client("http://0.0.0.0:6735")
        model = client.get_model("IndexTTS2")

        with open("../mp3_test_voice.mp3", "rb") as f:
            test_prompt_speech = f.read()

        response = model.speech(
            input="Quick, hide! He's coming! He's coming to get us!",
            prompt_speech=test_prompt_speech,
            emo_alpha=0.6,
            use_emo_text=True,
            emo_text="You scared the hell out of me! Are you a ghost?",
            use_random=False
        )

IndexTTS-2.5 Usage
~~~~~~~~~~~~~~~~~~

``IndexTTS-2.5`` adds Japanese, Spanish, and Arabic, as well as pronunciation
and speaking-speed control. A reference audio file and a language are used for
each request. Supported language values are ``ZH``, ``EN``, ``JA``, ``ES``, and
``AR``:

.. code-block:: python

    from xinference.client import Client

    client = Client("http://0.0.0.0:6735")
    model = client.get_model("IndexTTS-2.5")

    with open("prompt.wav", "rb") as f:
        prompt_speech = f.read()

    response = model.speech(
        input="Hello, this is a voice cloning demo.",
        prompt_speech=prompt_speech,
        language="EN",
        speed=1.2,
    )

The standard ``speed`` parameter uses OpenAI semantics: values above ``1.0``
speed up speech. The supported range is ``0.5`` to ``2.0``. You can instead
pass the upstream ``duration_factor`` option with the same supported range,
where values above ``1.0`` slow speech down. Pronunciation can be controlled
with ``<word|reading>`` notation, for example
``他在银<行|XING2>里<行|HANG2>走了半天。``.

Upstream officially supports Python 3.10 and 3.11. A GPU is strongly
recommended; enable ``use_bf16`` at launch to reduce GPU memory use. Text-based
emotion guidance additionally requires ``use_qwen_emo=True``.

IndexTTS2 Offline Usage
~~~~~~~~~~~~~~~~~~~~~~~~

IndexTTS2 requires several small models that are downloaded automatically during initialization.
For offline environments, you can download these models to a single directory and specify the directory path.

**Easy Setup Method**

The simplest way to set up offline usage is to Use the `hf download` command to download the small model in advance:

.. code-block:: bash

    # Create your local models directory
    mkdir -p /path/to/small_models

    # Download models from Hugging Face
    hf download facebook/w2v-bert-2.0 --local-dir /path/to/small_models/w2v-bert-2.0
    hf download funasr/campplus --local-dir /path/to/small_models/campplus
    hf download nvidia/bigvgan_v2_22khz_80band_256x --local-dir /path/to/small_models/bigvgan
    hf download amphion/MaskGCT --local-dir /path/to/small_models/MaskGCT

The final directory structure should look like this:

.. code-block:: text

    /path/to/small_models/
    ├── w2v-bert-2.0/                 # Feature extraction model
    ├── campplus/                     # Speaker recognition model
    ├── bigvgan/                      # Vocoder model
    └── MaskGCT/                      # Semantic codec model

**Required Models**

The small models are automatically mapped as follows:

1. **w2v-bert-2.0** (``models--facebook--w2v-bert-2.0``) - Feature extraction model
2. **campplus** (``models--funasr--campplus``) - Speaker recognition model
3. **bigvgan** (``models--nvidia--bigvgan_v2_22khz_80band_256x``) - Vocoder model
4. **semantic_codec** (``models--amphion--MaskGCT``) - Semantic encoding/decoding model

**Launching IndexTTS2 with Offline Models**

When launching IndexTTS2 with Web UI, you can add an additional parameter:
- ``small_models_dir`` - Path to directory containing all small models

When launching with command line, you can add the option:

.. code-block:: bash

    xinference launch --model-name IndexTTS2 --model-type audio \
        --small_models_dir /path/to/small_models

When launching with Python client:

.. code-block:: python

    model_uid = client.launch_model(
        model_name="IndexTTS2",
        model_type="audio",
        small_models_dir="/path/to/small_models"
    )

---

### Doc Source Models Model Abilities Chat (`doc/source/models/model_abilities/chat.rst`)

.. _chat:

=====================
Chat & Generate
=====================

Learn how to chat with LLMs in Xinference.

Introduction
============

Models equipped with ``chat`` or ``generate`` abilities are frequently referred to as large language models (LLM) or text generation models.
These models are designed to respond with text outputs to the inputs they receive, commonly known as "prompts".
Typically, one can direct these models using specific instructions or by providing concrete examples illustrating
how to accomplish a task.

Models with ``generate`` capacities are typically pre-trained large language models. On the other hand, models equipped with ``chat``
capabilities are finely-tuned and aligned LLMs, optimized for dialogues use case. In most cases, models ending with "chat" 
(e.g. ``llama-2-chat``, ``qwen-chat``, etc) are identified as having ``chat`` capabilities. 


The Chat API and Generate API offer two distinct approaches for interacting with LLMs:

* The Chat API (like OpenAI's `Chat Completion API <https://platform.openai.com/docs/api-reference/chat/create>`__)
  can conduct multi-turn conversations.

* The Generate API (like OpenAI's legacy `Completions API <https://platform.openai.com/docs/api-reference/completions/create>`__)
  allows you to generate text based on a text prompt.

.. list-table:: 
   :widths: 25 25 50
   :header-rows: 1

   * - MODEL ABILITY
     - API ENDPOINT
     - OpenAI-compatible ENDPOINT

   * - chat
     - Chat API
     - /v1/chat/completions

   * - generate
     - Generate API
     - /v1/completions


Supported models
-------------------

You can examine the abilities of all the :ref:`builtin LLM models in Xinference <models_llm_index>`.

Chat Models
===================

Chat API 
------------

The Chat API mimics OpenAI's `Chat Completion API <https://platform.openai.com/docs/api-reference/chat/create>`__. 
We can try Chat API out either via cURL, OpenAI Client, or Xinference's python client:

.. tabs::

  .. code-tab:: bash cURL

    curl -X 'POST' \
      'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/chat/completions' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "<MODEL_UID>",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "What is the largest animal?"
            }
        ],
        "max_tokens": 512,
        "temperature": 0.7        
      }'

  .. code-tab:: python OpenAI Python Client

    import openai

    client = openai.Client(
        api_key="cannot be empty", 
        base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1"
    )
    client.chat.completions.create(
        model="<MODEL_UID>",
        messages=[
            {
                "content": "What is the largest animal?",
                "role": "user",
            }
        ],
        max_tokens=512,
        temperature=0.7
    )

  .. code-tab:: python Xinference Python Client

    from xinference.client import RESTfulClient

    client = RESTfulClient("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
    model = client.get_model("<MODEL_UID>")
    messages = [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the largest animal?"}]
    model.chat(
        messages,
        generate_config={
          "max_tokens": 512,
          "temperature": 0.7
        }        
    )

  .. code-tab:: json output

    {
      "id": "chatcmpl-8d76b65a-bad0-42ef-912d-4a0533d90d61",
      "model": "<MODEL_UID>",
      "object": "chat.completion",
      "created": 1688919187,
      "choices": [
        {
          "index": 0,
          "message": {
            "role": "assistant",
            "content": "The largest animal that has been scientifically measured is the blue whale, which has a maximum length of around 23 meters (75 feet) for adult animals and can weigh up to 150,000 pounds (68,000 kg). However, it is important to note that this is just an estimate and that the largest animal known to science may be larger still. Some scientists believe that the largest animals may not have a clear \"size\" in the same way that humans do, as their size can vary depending on the environment and the stage of their life."
          },
          "finish_reason": "None"
        }
      ],
      "usage": {
        "prompt_tokens": -1,
        "completion_tokens": -1,
        "total_tokens": -1
      }
    }


Hybrid Thinking Models
----------------------

Some LLMs are marked as ``hybrid`` and can run with or without thinking mode.

.. versionadded:: v1.17.0
  Request-level ``enable_thinking`` is added in v1.17.0

Xinference exposes a request-level ``enable_thinking`` switch that works across different model templates (e.g. Qwen
uses ``enable_thinking`` while some DeepSeek templates use ``thinking``).

Usage examples:

.. tabs::

  .. code-tab:: bash cURL

    curl -X 'POST' \
      'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/chat/completions' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "<MODEL_UID>",
        "messages": [
            {"role": "user", "content": "What is the largest animal?"}
        ],
        "enable_thinking": false
      }'

  .. code-tab:: python OpenAI Python Client

    import openai

    client = openai.Client(
        api_key="cannot be empty",
        base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1"
    )
    client.chat.completions.create(
        model="<MODEL_UID>",
        messages=[
            {"role": "user", "content": "What is the largest animal?"}
        ],
        extra_body={"enable_thinking": False}
    )

  .. code-tab:: python Xinference Python Client

    from xinference.client import RESTfulClient

    client = RESTfulClient("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
    model = client.get_model("<MODEL_UID>")
    model.chat(
        [{"role": "user", "content": "What is the largest animal?"}],
        enable_thinking=False,
    )

  .. code-tab:: python Xinference Python Client (explicit chat_template_kwargs)

    model.chat(
        [{"role": "user", "content": "What is the largest animal?"}],
        generate_config={"chat_template_kwargs": {"enable_thinking": False}},
    )


DeepSeek-V4-Flash-0731
~~~~~~~~~~~~~~~~~~~~~~

``DeepSeek-V4-Flash-0731`` is a separate built-in model from the
``DeepSeek-V4-Flash`` preview entry. It uses the native FP8 checkpoint and
requires vLLM 0.20.1 or newer.

Launch it with:

.. code-block:: bash

   xinference launch \
       --model-engine vLLM \
       --model-name DeepSeek-V4-Flash-0731 \
       --size-in-billions 304 \
       --model-format fp8 \
       --quantization fp8

The upstream repository supplies DeepSeek-V4-specific encoding code. Enable
trusted repository code before launch:

.. code-block:: bash

   export XINFERENCE_TRUST_REMOTE_CODE=1

The ``chat_template_kwargs`` option selects thinking mode and reasoning level:

.. code-block:: json

   {
     "chat_template_kwargs": {
       "enable_thinking": true,
       "reasoning_effort": "high"
     }
   }

Set ``enable_thinking`` to ``false`` for chat mode. Supported reasoning levels
are provided by the model repository; ``low``, ``high``, and ``max`` are
forwarded without Xinference rewriting them.

The checkpoint includes a DSpark speculative decoding module, but Xinference
does not enable it automatically. Enable it through vLLM model configuration
when supported:

.. code-block:: json

   {
     "speculative_config": {
       "method": "dspark",
       "num_speculative_tokens": 7,
       "draft_sample_method": "greedy"
     }
   }

The following parameters are the upstream example for a single 4xGB300 node
and are not Xinference defaults:

.. code-block:: bash

   --kv-cache-dtype fp8 \
   --block-size 256 \
   --data-parallel-size 4 \
   --enable-expert-parallel \
   --moe-backend deep_gemm_mega_moe \
   --attention-config '{"use_fp4_indexer_cache": true}'


Generate Models
================

Generate API
-------------

The Generate API mirrors OpenAI's legacy `Completions API <https://platform.openai.com/docs/api-reference/completions/create>`__.

The difference between the Generate API and the Chat API lies primarily in the form of input. Opposite to the Chat API that takes
a list of messages as input, the Generate API accepts a freeform text string named "prompt".

.. tabs::

  .. code-tab:: bash cURL

    curl -X 'POST' \
      'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/completions' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "<MODEL_UID>",
        "prompt": "What is the largest animal?",
        "max_tokens": 512,
        "temperature": 0.7
      }'

  .. code-tab:: python OpenAI Python Client

    import openai

    client = openai.Client(api_key="cannot be empty", base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1")
    client.chat.completions.create(
        model=("<MODEL_UID>",
        messages=[
            {"role": "user", "content": "What is the largest animal?"}
        ],
        max_tokens=512,
        temperature=0.7
    )

  .. code-tab:: python Xinference Python Client

    from xinference.client import RESTfulClient

    client = RESTfulClient("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
    model = client.get_model("<MODEL_UID>")
    print(model.generate(
        prompt="What is the largest animal?",
        generate_config={
          "max_tokens": 512,
          "temperature": 0.7
        }
    ))

  .. code-tab:: json output

    {
      "id": "cmpl-8d76b65a-bad0-42ef-912d-4a0533d90d61",
      "model": "<MODEL_UID>",
      "object": "text_completion",
      "created": 1688919187,
      "choices": [
        {
          "index": 0,
          "text": "The largest animal that has been scientifically measured is the blue whale, which has a maximum length of around 23 meters (75 feet) for adult animals and can weigh up to 150,000 pounds (68,000 kg). However, it is important to note that this is just an estimate and that the largest animal known to science may be larger still. Some scientists believe that the largest animals may not have a clear \"size\" in the same way that humans do, as their size can vary depending on the environment and the stage of their life.",
          "finish_reason": "None"
        }
      ],
      "usage": {
        "prompt_tokens": -1,
        "completion_tokens": -1,
        "total_tokens": -1
      }
    }




FAQ
========

Does Xinference's LLM provide integration methods for LangChain or LlamaIndex?
-----------------------------------------------------------------------------------

Yes, you can refer to the related sections in their respective official Xinference documentation. Here are the links:

* `LangChain LLMs: Xinference <https://reference.langchain.com/python/langchain-community/llms/xinference/Xinference>`__

* `LlamaIndex LLM integrations: Xinference  <https://docs.llamaindex.ai/en/stable/examples/llm/xinference_local_deployment.html>`__

---

### Doc Source Models Model Abilities Embed (`doc/source/models/model_abilities/embed.rst`)

.. _embed:

=====================
Embeddings
=====================


Learn how to create text embeddings in Xinference.


Introduction
============

Text embeddings are used to quantify how related different pieces of text are. They can be used in various applications including
search, clustering, recommendations, anomaly detection, diversity measurement, and classification.

An embedding is a vector of floating point numbers. The proximity between two vectors serves as an indicator of their similarity. 
Less distance implies a higher correlation, while a larger distance indicates reduced correlation.

Embedding models in Xinference can be invoked through the Embeddings API to create embeddings. 
The Embeddings API mimics OpenAI's `create embeddings API <https://platform.openai.com/docs/api-reference/embeddings/create>`_.

.. list-table:: 
   :widths: 25 50
   :header-rows: 1

   * - API ENDPOINT
     - OpenAI-compatible ENDPOINT

   * - Embeddings API
     - /v1/embeddings


Supported models
-------------------

You can examine all the :ref:`builtin embedding models in Xinference <models_embedding_index>`.

Embedding engines
-------------------

When launching an embedding model, you can pick the serving engine with the
``model_engine`` parameter (``--model-engine`` on the command line):

* ``sentence_transformers``: the default engine, available for all embedding
  models.
* ``vllm``: high-throughput serving for supported model families — currently
  models whose names start with ``bge``, ``gte``, ``text2vec``, ``m3e``,
  ``Qwen3``, ``WeMM``, or ``bce`` (e.g. ``bce-embedding-base_v1``).
* ``flag``: FlagEmbedding-based engine; also supports hybrid (sparse+dense)
  output, see the FAQ below.
* ``llama.cpp``: serve GGUF-format embedding models.

Truncating input
-------------------

The Embeddings API accepts an optional ``truncate_prompt_tokens`` parameter
to cap the token length of each input before encoding:

* unset / ``null``: no truncation.
* a positive integer ``N``: truncate each input to at most ``N`` tokens.
* ``-1``: truncate to the model's own maximum input length.

.. code-block:: bash

    curl -X 'POST' \
      'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/embeddings' \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "<MODEL_UID>",
        "input": "A very long document ...",
        "truncate_prompt_tokens": 512
      }'


Multimodal WeMM-Embedding input
-------------------------------

The WeMM-Embedding family accepts text, image, video, visual-document, and
interleaved multimodal input. Audio is not supported. Supported engines are
``sentence_transformers`` (default) and ``vllm``.

A flat input dictionary can contain any ordered combination of ``text``,
``image``, and ``video``. For precise interleaving, pass a ``role``/``content``
message. ``image_url`` and ``video_url`` content items are also accepted.
Local media paths refer to files on the Xinference server.

.. code-block:: bash

    xinference launch \
      --model-name WeMM-Embedding-2B \
      --model-type embedding \
      --model-engine sentence_transformers

    curl -X POST \
      'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/embeddings' \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "<MODEL_UID>",
        "input": {
          "role": "user",
          "content": [
            {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
            {"type": "text", "text": "Find content related to this image."},
            {"type": "video_url", "video_url": {"url": "https://example.com/video.mp4"}}
          ]
        },
        "dimensions": 256
      }'

``dimensions`` selects one of the model's Matryoshka dimensions. By default,
Xinference truncates the vector and applies L2 normalization again.


Quickstart
============

We can try Embeddings API out either via cURL, OpenAI Client, or Xinference's python client:

.. tabs::

  .. code-tab:: bash cURL

    curl -X 'POST' \
      'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/embeddings' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "<MODEL_UID>",
        "input": "What is the capital of China?"
      }'

  .. code-tab:: python OpenAI Python Client

    import openai

    client = openai.Client(
      api_key="cannot be empty", 
      base_url="http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1"
    )
    client.embeddings.create(
      model=model_uid, 
      input=["What is the capital of China?"]
    )

  .. code-tab:: python Xinference Python Client

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

    model = client.get_model("<MODEL_UID>")
    input = "What is the capital of China?"
    model.create_embedding(input)

  .. code-tab:: json output

    {
      "object": "list",
      "model": "<MODEL_UID>",
      "data": [{
        "index": 0,
        "object": "embedding",
        "embedding": [
          -0.014207549393177032, 
          -0.01832585781812668, 
          ...
          -0.03009396605193615,
          0.05420297756791115]
      }],
      "usage": {
        "prompt_tokens": 37,
        "total_tokens": 37
      }
    }


FAQ
========

Does the LLM in Xinference support Embeddings API?
------------------------------------------------------------

No. Xinference doesn't provide embed API for LLMs due to considerations of performance.


Does Embeddings API provides integration method for LangChain?
-----------------------------------------------------------------------------------

Yes, you can refer to the related sections in LangChain's respective official Xinference documentation.
Here is the link: `Text Embedding Models: Xinference <https://reference.langchain.com/python/langchain-community/embeddings/xinference/XinferenceEmbeddings>`_ 


Does Embeddings API support hrbrid model?
-----------------------------------------------------------------------------------

Yes, you can use ``flag`` as the engine to deploy the model and call Embeddings API by setting the extra parameter ``return_parse=True`` which will return sparse vectors.

---

### Doc Source Models Model Abilities Flexible (`doc/source/models/model_abilities/flexible.rst`)

.. _flexible:

=====================
Traditional ML models
=====================

Learn how to inference traditional machine learning models with Xinference.
These flexibly extensible models are referred to as **Flexible Models** within Xinference.

.. versionadded:: v1.7.1


Introduction
==================

Traditional machine learning models can still play a significant role within an LLM-centric ecosystem.

Xinference provides flexible extensibility for performing inference with traditional machine learning models.
It includes built-in support for loading and running the following types of models:

- HuggingFace Pipelines for tasks such as classification using models hosted on HuggingFace.
- ModelScope Pipelines for tasks such as classification using models from ModelScope.
- YOLO for image detection and related computer vision tasks.

A wide range of traditional machine learning models can be used with Xinference.
For each of the categories above, we will walk through a representative example to
demonstrate how to perform inference step by step on the Xinference platform.

Built-in Model Support Examples
================================

HuggingFace Pipeline Model
~~~~~~~~~~~~~~~~~~~~~~~~~~

First, we use `FacebookAI/roberta-large-mnli <https://huggingface.co/FacebookAI/roberta-large-mnli>`_ as an example.
This is a zero-shot classification model.
For other types of models, simply specify the corresponding task (which is also a parameter of the Pipeline)
when registering the model.

Download the model to the following path::

    /path/to/roberta-large-mnli

Next, we demonstrate how to register this flexible model in the Xinference Web UI.
For the following examples, unless we have to, we will skip the UI steps and focus on the core logic.

.. raw:: html

    

The corresponding custom model JSON file is as follows:

.. code-block:: json

    {
        "model_name": "roberta-large-mnli",
        "model_id": null,
        "model_revision": null,
        "model_hub": "huggingface",
        "model_description": "roberta-large-mnli is the RoBERTa large model fine-tuned on the Multi-Genre Natural Language Inference (MNLI) corpus. The model is a pretrained model on English language text using a masked language modeling (MLM) objective.",
        "model_uri": "/path/to/roberta-large-mnli",
        "launcher": "xinference.model.flexible.launchers.transformers",
        "launcher_args": "{\"task\": \"zero-shot-classification\"}",
        "virtualenv": {
            "packages": [],
            "inherit_pip_config": true,
            "index_url": null,
            "extra_index_url": null,
            "find_links": null,
            "trusted_host": null,
            "no_build_isolation": null
        },
        "is_builtin": false
    }

Refer to the section :ref:`register_custom_model` for instructions on registering the model using either code or the command line.

Next, load the model by selecting **Launch Model** / **Custom Model** / **Flexible Model** in the Web UI.
The loading procedure is the same as for other model types.

When using the command line, remember to specify the option ``--model-type flexible``.

After the model is successfully loaded, we can perform inference using the following method.

.. tabs::

  .. code-tab:: bash cURL

    curl -X 'POST' \
      'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/flexible/infers' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "roberta-large-mnli",
        "args": [
          "one day I will see the world",
          ["travel", "cooking", "dancing"]
        ]
      }'

  .. code-tab:: python Xinference Python Client

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

    model = client.get_model("roberta-large-mnli")

    sequence_to_classify = "one day I will see the world"
    candidate_labels = ['travel', 'cooking', 'dancing']
    model.infer(sequence_to_classify, candidate_labels)


  .. code-tab:: json output

    {"sequence":"one day I will see the world","labels":["travel","cooking","dancing"],"scores":[0.9799638986587524,0.010605016723275185,0.009431036189198494]}

Local text classification models
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Register a local BERT or other Transformers sequence classification model using
the same launcher with ``task="text-classification"``. The directory must contain
a trained classification model and tokenizer saved with ``save_pretrained``;
a base encoder or a training checkpoint alone is insufficient. The model's
``config.json`` supplies ``id2label``, ``label2id`` and ``problem_type``.
The directory must be accessible on the worker that loads the model.

ModelScope Pipeline Model
~~~~~~~~~~~~~~~~~~~~~~~~~

ModelScope Pipeline models are very similar to Huggingface ones.
The only difference lies in the launcher used.

We take a zero-shot classification model from ModelScope as an example.
The model is `iic/nlp_structbert_zero-shot-classification_chinese-base <https://modelscope.cn/models/iic/nlp_structbert_zero-shot-classification_chinese-base>`_.

Here, we make use of Xinference's model virtual environment feature.
This is because the model used in this example requires ``transformers==4.50.3`` to run properly.
To isolate the environment, we use a :ref:`virtual env <model_virtual_env>` when registering the model.

When specifying custom packages during registration, the syntax is the same as for regular packages, with a few special cases.
Since the virtual environment is still based on the site packages of the Python runtime where Xinference is running, we need to explicitly include ``#system_numpy#``.
Packages wrapped in ``#system_xx#`` ensure consistency with the base environment during virtual environment creation; otherwise, it may easily result in runtime errors.

Registering via Web UI:

.. raw:: html

    

Corresponding json file:

.. code-block:: json

    {
        "model_name": "nlp_structbert_zero-shot-classification_chinese-base",
        "model_id": null,
        "model_revision": null,
        "model_hub": "huggingface",
        "model_description": "This is a model description.",
        "model_uri": "/Users/xuyeqin/Downloads/models/nlp_structbert_zero-shot-classification_chinese-base",
        "launcher": "xinference.model.flexible.launchers.modelscope",
        "launcher_args": "{\"task\": \"zero-shot-classification\"}",
        "virtualenv": {
            "packages": [
                "transformers==4.50.3",
                "#system_numpy#"
            ],
            "inherit_pip_config": true,
            "index_url": "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple",
            "extra_index_url": null,
            "find_links": null,
            "trusted_host": null,
            "no_build_isolation": null
        },
        "is_builtin": false
    }

Inference the model:

.. tabs::

  .. code-tab:: bash cURL

    curl -X 'POST' \
      'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/flexible/infers' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
        "model": "nlp_structbert_zero-shot-classification_chinese-base",
        "args": [
          "世界那么大，我想去看看"
        ],
        "candidate_labels": ["家居", "旅游", "科技", "军事", "游戏", "故事"]
      }'

  .. code-tab:: python Xinference Python Client

    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")

    model = client.get_model("nlp_structbert_zero-shot-classification_chinese-base")

    labels = ['家居', '旅游', '科技', '军事', '游戏', '故事']
    sentence = '世界那么大，我想去看看'
    model.infer(sentence, candidate_labels=labels)


  .. code-tab:: json output

    {"labels":["旅游","故事","游戏","家居","科技","军事"],"scores":[0.5115892291069031,0.1660086065530777,0.11971458047628403,0.08431519567966461,0.06298774480819702,0.05538458004593849]}%

YOLO
~~~~

YOLO is a popular real-time object detection model, widely used in image detection and video analysis scenarios.

First, download the YOLO weights.
Here, we use the `yolov11s.pt <https://huggingface.co/Ultralytics/YOLO11>`_ file as an example.

JSON file of model definition:

.. code-block:: json

    {
        "model_name": "yolo11s",
        "model_id": null,
        "model_revision": null,
        "model_hub": "huggingface",
        "model_description": "YOLO is a popular real-time object detection model, widely used in image detection and video analysis scenarios.",
        "model_uri": "/Users/xuyeqin/Downloads/models/yolo11s.pt",
        "launcher": "xinference.model.flexible.launchers.yolo",
        "launcher_args": "{}",
        "virtualenv": {
            "packages": [
                "ultralytics",
                "#system_numpy#"
            ],
            "inherit_pip_config": true,
            "index_url": "https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple",
            "extra_index_url": null,
            "find_links": null,
            "trusted_host": null,
            "no_build_isolation": null
        },
        "is_builtin": false
    }

Inference the model:

.. tabs::

  .. code-tab:: python Xinference Python Client

    import requests
    from PIL import Image
    import io
    import base64
    from xinference.client import Client

    client = Client("http://<XINFERENCE_HOST>:<XINFERENCE_PORT>")
    model = client.get_model("yolo11s")

    url = "https://ultralytics.com/images/bus.jpg"

    response = requests.get(url)
    response.raise_for_status()

    img = Image.open(io.BytesIO(response.content))

    buffered = io.BytesIO()
    img.save(buffered, format="JPEG")
    img_bytes = buffered.getvalue()
    img_base64 = base64.b64encode(img_bytes).decode('utf-8')

    model.infer(source=img_base64)

  .. code-tab:: json output

    [[{'name': 'bus',
       'class': 5,
       'confidence': 0.93653,
       'box': {'x1': 13.9521, 'y1': 227.0665, 'x2': 800.17688, 'y2': 739.13965}},
      {'name': 'person',
       'class': 0,
       'confidence': 0.89741,
       'box': {'x1': 669.89709,
        'y1': 389.82065,
        'x2': 809.58966,
        'y2': 879.65491}},
      {'name': 'person',
       'class': 0,
       'confidence': 0.88205,
       'box': {'x1': 52.37262, 'y1': 397.83792, 'x2': 248.506, 'y2': 905.98212}},
      {'name': 'person',
       'class': 0,
       'confidence': 0.8706,
       'box': {'x1': 222.58685,
        'y1': 405.93442,
        'x2': 345.02032,
        'y2': 859.52789}},
      {'name': 'person',
       'class': 0,
       'confidence': 0.66505,
       'box': {'x1': 0.28522, 'y1': 548.60931, 'x2': 81.25904, 'y2': 871.59076}}]]

Writing a Custom Flexible Model
==================================

First, we implement a custom launcher with a simple model for sentiment scoring.
In this example, we do not use any actual model weights, so the ``load`` function does not perform any model loading.

.. code-block:: python

    # my_flexible_model.py

    from xinference.model.flexible import FlexibleModel


    class RuleBasedSentimentModel(FlexibleModel):
        def load(self):
            self.pos_words = self.config.get("pos", ["good", "happy", "great"])
            self.neg_words = self.config.get("neg", ["bad", "sad", "terrible"])

        def infer(self, text: str):
            score = 0
            words = text.lower().split()
            for w in words:
                if w in self.pos_words:
                    score += 1
                elif w in self.neg_words:
                    score -= 1
            return {"score": score}


    def launcher(model_uid: str, model_spec: FlexibleModel, **kwargs) -> FlexibleModel:
        # get model path,
        # in this example, we do not use it, so it's empty
        model_path = model_spec.model_uri
        return RuleBasedSentimentModel(model_uid=model_uid, model_path=model_path, config=kwargs)

The model JSON definition is as follows:

.. code-block:: json

    {
        "model_name": "my-flexible-model",
        "model_id": null,
        "model_revision": null,
        "model_hub": "huggingface",
        "model_description": "This is a model description.",
        "model_uri": "/path/to/model",
        "launcher": "my_flexible_model.launcher",
        "launcher_args": "{\"pos\": [\"good\", \"happy\", \"great\", \"nice\"]}",
        "virtualenv": {
            "packages": [],
            "inherit_pip_config": true,
            "index_url": null,
            "extra_index_url": null,
            "find_links": null,
            "trusted_host": null,
            "no_build_isolation": null
        },
        "is_builtin": false
    }

Here, we extend the model by passing in a custom-defined ``pos`` value.

Finally, let's verify the result:

.. tabs::

  .. code-tab:: python Xinference Python Client

    from xinference.client import Client

    client = Client("http://127.0.0.1:9997")

    model = client.get_model("my-flexible-model")

    model.infer("I feel nice and am happy today")

  .. code-tab:: json output

    {'score': 2}

Conclusion
==================

The built-in Flexible Model launchers in Xinference can be found at
`Github <https://github.com/xorbitsai/inference/tree/main/xinference/model/flexible/launchers>`_.
Contributions are welcome to support more traditional machine learning models!

---

### Doc Source Models Model Abilities Image (`doc/source/models/model_abilities/image.rst`)

.. _image:

========
Images
========

Learn how to generate images with Xinference.


Introduction
==================


The Images API provides two methods for interacting with images:


* The Text-to-image endpoint create images from scratch based on a text prompt.
* The Image-to-image endpoint allows you to generate a variation of a given image.


.. list-table:: 
   :widths: 25  50
   :header-rows: 1

   * - API ENDPOINT
     - OpenAI-compatible ENDPOINT

   * - Text-to-Image API
     - /v1/images/generations

   * - Image-to-image API
     - /v1/images/variations

Supported models
-------------------

The Text-to-image API is supported with the following models in Xinference:

* sd-turbo
* sdxl-turbo
* stable-diffusion-v1.5
* stable-diffusion-xl-base-1.0
* sd3-medium
* sd3.5-medium
* sd3.5-large
* sd3.5-large-turbo
* FLUX.1-schnell
* FLUX.1-dev
* Kolors
* hunyuandit-v1.2
* hunyuandit-v1.2-distilled
* cogview4
* Qwen-Image
* GLM-Image
* Ideogram4
* HiDream-O1-Image
* HiDream-O1-Image-Dev
* HiDream-O1-Image-Dev-2604
* Krea-2-Raw
* Krea-2-Turbo

Image-to-image supported models:

* Flux.1-Kontext-dev
* Qwen-Image-Edit
* GLM-Image
* HiDream-O1-Image
* HiDream-O1-Image-Dev


HiDream-O1 runtime notes
------------------------

All three HiDream-O1 checkpoints are available from Hugging Face with revision
``main`` and ModelScope with revision ``master``. They require an NVIDIA CUDA
GPU and PyTorch 2.10 or newer. Flash Attention is optional; Xinference disables
it by default for compatibility.

.. list-table::
   :widths: 30 15 55
   :header-rows: 1

   * - Model
     - Default steps
     - Usage and scheduler defaults
   * - HiDream-O1-Image
     - 50
     - Text-to-image, instruction-based editing, and multiple reference images;
       uses the official default scheduler.
   * - HiDream-O1-Image-Dev
     - 28
     - Text-to-image, editing, and multiple reference images; single-reference
       editing uses the flow-matching scheduler, while text-to-image and
       multi-reference generation use the flash scheduler.
   * - HiDream-O1-Image-Dev-2604
     - 28
     - Text-to-image only; uses the checkpoint's float32 loading and sampling
       defaults.


Image engines
-------------------

Text-to-image models run on the ``diffusers`` engine by default. On Linux with
NVIDIA GPUs, the following models can also run on the ``SGLang`` engine
(powered by `sglang-diffusion <https://docs.sglang.io/docs/sglang-diffusion/installation>`_)
or the ``vLLM`` engine
(powered by `vllm-omni <https://docs.vllm.ai/projects/vllm-omni/>`_)
for faster inference:

* FLUX.1-dev
* GLM-Image (SGLang only)
* Krea-2-Raw (SGLang only)
* Krea-2-Turbo (SGLang only)
* Qwen-Image
* Qwen-Image-2512
* Z-Image
* Z-Image-Turbo
* sd3.5-medium (vLLM only)

On the vLLM engine, models whose vllm-omni pipeline supports request-level
batching (e.g. sd3.5-medium, Qwen-Image, FLUX.1-dev) can additionally batch
concurrent requests on the GPU; pass ``max_num_seqs`` at launch to set the
maximum batch size.

SGLang 0.5.20 or newer also accepts concurrent image requests for native dynamic batching. Set ``batching_max_size`` and optionally ``batching_delay_ms`` at launch. For vLLM-Omni 0.28, set ``max_num_seqs``; supported pipelines can additionally use ``step_execution=true`` for experimental continuous batching. Batching is opt-in, requires compatible requests and model support, and increases GPU memory usage. Send independent API requests concurrently; ``n`` only controls the number of outputs within one request.

To use them, install SGLang with diffusion support via
``pip install 'sglang[diffusion]>=0.5.20,<0.6'``, or vLLM-Omni together with a vLLM of the
same major.minor version via ``pip install 'vllm-omni==0.28.*' 'vllm==0.28.*'``,
then launch the model with ``--model-engine SGLang`` or
``--model-engine vLLM``, for example:

.. code-block:: bash

    xinference launch --model-name Z-Image-Turbo --model-type image --model-engine SGLang
    xinference launch --model-name Z-Image-Turbo --model-type image --model-engine vLLM

Note that GGUF quantization, Lightning acceleration, LoRA and controlnet are
only available on the ``diffusers`` engine.


Ideogram4
-------------------

:ref:`Ideogram4 <models_builtin_ideogram4>` uses the NF4 checkpoint and
requires an NVIDIA CUDA GPU. The checkpoint is distributed under the Ideogram
4 Non-Commercial Model Agreement, and its repositories are gated. Accept the
license and authenticate with the selected model hub before launching.

Ideogram4 accepts plain text prompts, but serialized structured JSON captions
provide the best quality and control.

To download from ModelScope for an individual launch::

   xinference launch --model-name Ideogram4 --model-type image --download_hub modelscope


GLM-Image
-------------------

:ref:`GLM-Image <models_builtin_glm-image>` supports both text-to-image
generation and single- or multi-reference image-to-image generation through
the same ``GlmImagePipeline``. Output width and height must both be divisible
by 32.

The ``diffusers`` engine uses ``diffusers==0.38.0`` and
``transformers==5.0.0``. Xinference installs these packages automatically when
per-model virtual environments are enabled.

The Hugging Face source is ``zai-org/GLM-Image`` at revision ``main``. The
ModelScope source is ``ZhipuAI/GLM-Image`` at revision ``master``.

To download from ModelScope for an individual launch::

   xinference launch --model-name GLM-Image --model-type image --download_hub modelscope

On Linux with NVIDIA GPUs, text-to-image generation can also use SGLang::

   xinference launch --model-name GLM-Image --model-type image --model-engine SGLang

The SGLang engine currently exposes only text-to-image generation. Use the
default ``diffusers`` engine for image-to-image generation.


Quickstart
===================

Text-to-image
--------------------

The Text-to-image API mimics OpenAI's `create images API <https://platform.openai.com/docs/api-reference/images/create>`_.
We can try Text-to-image API out either via cURL, OpenAI Client, or Xinference's python client:

.. tabs::

  .. code-tab:: bash cURL

    curl -X 'POST' \
      'http://<XINFERENCE_HOST>:<XINFERENCE_PORT>/v1/images/generations' \
      -H 'accept: application/json' \