mlc-llm Documentation

File: README.md

About

MLC LLM is a machine learning compiler and high-performance deployment engine for large language models. The mission of this project is to enable everyone to develop, optimize, and deploy AI models natively on everyone's platforms.

MLC LLM compiles and runs code on MLCEngine -- a unified high-performance LLM inference engine across the above platforms. MLCEngine provides OpenAI-compatible API available through REST server, python, javascript, iOS, Android, all backed by the same engine and compiler that we keep improving with the community.

Get Started

Please visit our documentation to get started with MLC LLM.

Citation

Please consider citing our project if you find it useful:

bibtex

@software{mlc-llm,
    author = {{MLC team}},
    title = {{MLC-LLM}},
    url = {https://github.com/mlc-ai/mlc-llm},
    year = {2023-2025}
}

The underlying techniques of MLC LLM include:

References (Click to expand)

text

/* Detailed source-code truncated for AI context efficiency. */

File: android/MLCChat/README.md

MLC-LLM Android

Checkout Documentation page for more information.

run mlc_llm package
open this MLCChat/ folder as a project in Android Studio

File: android/MLCEngineExample/README.md

MLC-LLM Android

Checkout Documentation page for more information.

run mlc_llm package
open this MLCEngineExample/ folder as a project in Android Studio

File: android/README.md

MLC-LLM Android

Documentation page

File: docs/community/faq.rst

.. _FAQ:

Frequently Asked Questions

This is a list of Frequently Asked Questions (FAQ) about the MLC-LLM. Feel free to suggest new entries!

... How can I customize the temperature, and repetition penalty of models?
Please check our :ref:configure-mlc-chat-json tutorial.

... What's the quantization algorithm MLC-LLM using?
Please check our :doc:/compilation/configure_quantization tutorial.

... Why do I encounter an error free(): invalid pointer, Aborted (core dumped) at the end of model compilation?
This happens if you compiled TVM from source and didn't hide LLVM symbols in cmake configurations.
Please follow our instructions in :ref:Building TVM from Source <tvm-build-from-source> tutorial to compile TVM which hides LLVM symbols, or use our pre-built MLC-LLM :doc:pip wheels <../install/mlc_llm>.

File: docs/community/guideline.rst

.. _community_guide:

Community Guideline

.. contents::
:depth: 2
:local:

Welcome to the MLC-LLM community! Just like you, all of us are in awe of the immense power of large language models.
Our goal for MLC-LLM is to foster a project that is driven by an open-source community, working together to democratize
this technology and make it accessible across various devices. We are thrilled to have you as part of our
community and eagerly anticipate your valuable contributions.

.. _community_discussion:

Participate in Community Discussions

We encourage open discussions. If you encounter a bug or have a feature request, please file an issue in MLC-LLM's
GitHub issue tracker <https://github.com/mlc-ai/mlc-llm/issues>__. You are encouraged to tag the issue with labels
such as "bug," "feature request," or "iOS" so that the relevant developers can quickly notice your concern.

Additionally, we have set up a discord server <https://discord.gg/9Xpy2HGBuD>__ for online discussions.
While we encourage participation in the Discord server, we also recommend creating a GitHub issue even if the
topic has been discussed there. This ensures that the discussion is archived and searchable for future reference.

Before submitting an issue, we kindly ask you to check our :doc:/community/faq to see if your question has already been answered.

.. _contribute-to-mlc-llm:

Contribute to MLC-LLM

.. _fork-and-create-pull-requests:

Fork and Create Pull Requests
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Ready to contribute to MLC-LLM? Awesome! We are excited to see you are ready to contribute your code.
The standard way to make changes to MLC-LLM code base is through creating a pull-request <https://github.com/mlc-ai/mlc-llm/pulls>__,
and we will review your code and merge it to the code base when it is ready.

The first step to becoming a developer is to fork <https://github.com/mlc-ai/mlc-llm/fork>__ the repository to your own
github account, you will notice a repository under https://github.com/username/mlc-llm where username is your github user name.

You can clone your fork to your local machine and commit changes, or edit the contents of your fork (in the case you are just fixing typos)
on GitHub directly. Once your update is complete, you can click the contribute button and open a pull request to the main repository.

.. _contribute-new-models:

Contribute New Models to MLC-LLM
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If you have compiled a model using our :doc:/compilation/compile_models tutorial for an existing model architecture, please upload your models to the internet (e.g., Hugging Face) by following :ref:distribute-compiled-models tutorial.
If you add a new model variant to MLC-LLM by following our :doc:/compilation/define_new_models tutorial.
Please create a pull request to add your model architecture (currently model architectures are placed under
relax_models <https://github.com/mlc-ai/mlc-llm/tree/main/mlc_llm/relax_model>__ folder).

.. _coding-styles:

Coding Styles
^^^^^^^^^^^^^

For python codes, we generally follow the PEP8 style guide <https://peps.python.org/pep-0008/>.
The python comments follow NumPy style <https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html> python docstrings.
To make things easy, you can use black <https://pypi.org/project/black/>__ to automatically format your python code.

.. code:: bash

text

pip install black
  black your_python_file.py

For C++ codes, we generally follow the Google C++ style guide <https://google.github.io/styleguide/cppguide.html>.
The C++ comments should be Doxygen compatible <http://www.doxygen.nl/manual/docblocks.html#cppblock>.
Fo your convenience, you can use clang-format <https://clang.llvm.org/docs/ClangFormat.html>__ to automatically format your C++ code.

.. code:: bash

text

clang-format -i your_cpp_file.cpp

.. _general-development-process:

General Development Process

Everyone in the community is welcome to send patches, documents, and propose new directions to the project.
The key guideline here is to enable everyone in the community to get involved and participate in the decision and development.
We encourage public discussion in different channels, so that everyone in the community can participate
and get informed in developments.

Code reviews are one of the key ways to ensure the quality of the code. High-quality code reviews prevent technical debt
for long-term and are crucial to the success of the project. A pull request needs to be reviewed before it gets merged.
A committer who has the expertise of the corresponding area would moderate the pull request and merge the code when
it is ready. The corresponding committer could request multiple reviewers who are familiar with the area of the code.
We encourage contributors to request code reviews themselves and help review each other's code -- remember everyone
is volunteering their time to the community, high-quality code review itself costs as much as the actual code
contribution, you could get your code quickly reviewed if you do others the same favor.

The community should strive to reach a consensus on technical decisions through discussion. We expect committers to
moderate technical discussions in a diplomatic way, and provide suggestions with clear technical reasoning when necessary.

.. _roles-committers:

Committers
^^^^^^^^^^

Committers are individuals who are granted with write access to the project. A committer is usually responsible for
a certain area or several areas of the code where they oversee the code review process.
The area of contribution can take all forms, including code contributions and code reviews, documents, education, and outreach.
The review of pull requests will be assigned to the committers who recently contribute to the area this PR belongs to.
Committers are essential for a high quality and healthy project. The community actively looks for new committers
from contributors. Each existing committer can nominate new committers to MLC projects.

.. _roles-contributors:

Contributors
^^^^^^^^^^^^
We also welcome contributors if you are not ready to be a committer yet. Everyone who contributes to
the project (in the form of code, bugfix, documentation, tutorials, etc) is a contributor.
We maintain a page <https://github.com/mlc-ai/mlc-llm/blob/main/CONTRIBUTORS.md>__ to acknowledge contributors,
please let us know if you contribute to the project and if your name is not included in the list.

File: docs/compilation/compile_models.rst

.. _compile-model-libraries:

Compile Model Libraries

To run a model with MLC LLM in any platform, we need:

Model weights converted to MLC format (e.g. RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC <https://huggingface.co/mlc-ai/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/tree/main>__.)
Model library that comprises the inference logic

This page describes how to compile a model library with MLC LLM. Model compilation optimizes
the model inference for a given platform, allowing users bring their own new model
architecture, use different quantization modes, and customize the overall model
optimization flow.

Notably, in many cases you do not need to explicit call compile.

If you are using the Python API, you can skip specifying model_lib and
the system will JIT compile the library.
If you are building iOS/android package, checkout :ref:package-libraries-and-weights,
which provides a simpler high-level command that leverages the compile behind the scheme.

This page is still helpful to understand the compilation flow behind the scheme,
or be used to explicit create model libraries.
We compile RedPajama-INCITE-Chat-3B-v1 with q4f16_1 as an example for all platforms.

.. note::
Before you proceed, make sure you followed :ref:install-tvm, a required
backend to compile models with MLC LLM.

text

Please also follow the instructions in :ref:`deploy-cli` / :ref:`deploy-python-engine` to obtain
the CLI app / Python API that can be used to chat with the compiled model.

.. contents:: Table of Contents
:depth: 1
:local:

Verify Installation

Step 1. Verify mlc_llm

We use the python package mlc_llm to compile models. This can be installed by
following :ref:install-mlc-packages, either by building from source, or by
installing the prebuilt package. Verify mlc_llm installation in command line via:

.. code:: bash

text

$ mlc_llm --help
# You should see help information with this line
usage: MLC LLM Command Line Interface. [-h] {compile,convert_weight,gen_config}

.. note::
If it runs into error command not found: mlc_llm, try python -m mlc_llm --help.

Step 2. Verify TVM

To compile models, you also need to follow :ref:install-tvm.
Here we verify tvm quickly with command line (for full verification, see :ref:tvm-validate):

.. code:: bash

text

$ python -c "import tvm; print(tvm.__file__)"
/some-path/lib/python3.13/site-packages/tvm/__init__.py

Clone from HF and convert_weight

This replicates :ref:convert-weights-via-MLC, see that page for more details.

You can be under the mlc-llm repo, or your own working directory. Note that all platforms
can share the same compiled/quantized weights.

.. code:: shell

text

# Create directory
mkdir -p dist/models && cd dist/models
# Clone HF weights
git lfs install
git clone https://huggingface.co/togethercomputer/RedPajama-INCITE-Chat-3B-v1
cd ../..
# Convert weight
mlc_llm convert_weight ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \
    --quantization q4f16_1 \
    -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC

Generate mlc-chat-config and compile

A model library is specified by:

The model architecture (e.g. llama-2, gpt-neox)
Quantization (e.g. q4f16_1, q0f32)
Metadata (e.g. context_window_size, sliding_window_size, prefill-chunk-size), which affects memory planning
Platform (e.g. cuda, webgpu, iOS)

All these knobs are specified in mlc-chat-config.json generated by gen_config.

.. code:: shell

text

# Create output directory for the model library compiled
mkdir dist/libs

.. tabs::

text

.. group-tab:: Linux - CUDA

    .. code:: shell

        # 1. gen_config: generate mlc-chat-config.json and process tokenizers
        mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \
            --quantization q4f16_1 --conv-template redpajama_chat \
            -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
        # 2. compile: compile model library with specification in mlc-chat-config.json
        mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
            --device cuda -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-cuda.so


.. group-tab:: Metal

    For M-chip Mac:

    .. code:: shell

        # 1. gen_config: generate mlc-chat-config.json and process tokenizers
        mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \
            --quantization q4f16_1 --conv-template redpajama_chat \
            -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
        # 2. compile: compile model library with specification in mlc-chat-config.json
        mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
            --device metal -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-metal.so

    Cross-Compiling for Intel Mac on M-chip Mac:

    .. code:: shell

        # 1. gen_config: generate mlc-chat-config.json and process tokenizers
        mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \
            --quantization q4f16_1 --conv-template redpajama_chat \
            -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
        # 2. compile: compile model library with specification in mlc-chat-config.json
        mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
            --device metal:x86-64 -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-metal_x86_64.dylib

    For Intel Mac:

    .. code:: shell

        # 1. gen_config: generate mlc-chat-config.json and process tokenizers
        mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \
            --quantization q4f16_1 --conv-template redpajama_chat \
            -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
        # 2. compile: compile model library with specification in mlc-chat-config.json
        mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
            --device metal -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-metal_x86_64.dylib


.. group-tab:: Vulkan

    For Linux:

    .. code:: shell

        # 1. gen_config: generate mlc-chat-config.json and process tokenizers
        mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \
            --quantization q4f16_1 --conv-template redpajama_chat \
            -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
        # 2. compile: compile model library with specification in mlc-chat-config.json
        mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
            --device vulkan -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-vulkan.so

    For Windows:

    .. code:: shell

        # 1. gen_config: generate mlc-chat-config.json and process tokenizers
        mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \
            --quantization q4f16_1 --conv-template redpajama_chat \
            -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
        # 2. compile: compile model library with specification in mlc-chat-config.json
        mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
            --device vulkan -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-vulkan.dll

.. group-tab:: iOS/iPadOS

    You need a Mac to compile models for it.

    .. code:: shell

        # 1. gen_config: generate mlc-chat-config.json and process tokenizers
        mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ --quantization q4f16_1 \
            --conv-template redpajama_chat --context-window-size 768 \
            -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
        # 2. compile: compile model library with specification in mlc-chat-config.json
        mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
            --device iphone -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-iphone.tar

    .. note::
        If it runs into error

        .. code:: text

            Compilation error:
            xcrun: error: unable to find utility "metal", not a developer tool or in PATH
            xcrun: error: unable to find utility "metallib", not a developer tool or in PATH

        , please check and make sure you have Command Line Tools for Xcode installed correctly.
        You can use ``xcrun metal`` to validate: when it prints ``metal: error: no input files``, it means the Command Line Tools for Xcode is installed and can be found, and you can proceed with the model compiling.

.. group-tab:: Android

    .. code:: shell

        # 1. gen_config: generate mlc-chat-config.json and process tokenizers
        mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ --quantization q4f16_1 \
            --conv-template redpajama_chat --context-window-size 768 \
            -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
        # 2. compile: compile model library with specification in mlc-chat-config.json
        mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
            --device android -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-android.tar

.. group-tab:: WebGPU

    .. code:: shell

        # 1. gen_config: generate mlc-chat-config.json and process tokenizers
        mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \
            --quantization q4f16_1 --conv-template redpajama_chat \
            -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
        # 2. compile: compile model library with specification in mlc-chat-config.json
        mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
            --device webgpu -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-webgpu.wasm

    .. note::
        To compile for webgpu, you need to build from source when installing ``mlc_llm``. Besides, you also need to follow :ref:`install-web-build`.
        Otherwise, it would run into error

        .. code:: text

            RuntimeError: Cannot find libraries: wasm_runtime.bc

    .. note::
        For webgpu, when compiling larger models like ``Llama-2-7B``, you may want to add ``--prefill-chunk-size 1024`` or lower ``--context-window-size`` to decrease memory usage.
        Otherwise, you may run into issues like:

        .. code:: text

            TypeError: Failed to execute 'createBuffer' on 'GPUDevice': Failed to read the 'size' property from
            'GPUBufferDescriptor': Value is outside the 'unsigned long long' value range.

.. note::

text

For the ``conv-template``, `conversation_template.py <https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/conversation_template.py>`__
contains a full list of conversation templates that MLC provides. If the model you are adding
requires a new conversation template, you would need to add your own.
Follow `this PR <https://github.com/mlc-ai/mlc-llm/pull/2163>`__ as an example.
However, adding your own template would require you :ref:`build mlc_llm from source <mlcchat_build_from_source>`
in order for it to be recognized by the runtime.

For more details, please see :ref:`configure-mlc-chat-json`.

Verify output and chat

By executing the compile command above, we generate the model weights, model lib, and a chat config.
We can check the output with the commands below:

.. tabs::

text

.. group-tab:: Linux - CUDA

    .. code:: shell

        ~/mlc-llm > ls dist/libs
          RedPajama-INCITE-Chat-3B-v1-q4f16_1-cuda.so      # ===> the model library

        ~/mlc-llm > ls dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC
          mlc-chat-config.json                             # ===> the chat config
          tensor-cache.json                               # ===> the model weight info
          params_shard_0.bin                               # ===> the model weights
          params_shard_1.bin
          ...
          tokenizer.json                                   # ===> the tokenizer files
          tokenizer_config.json

    We can now chat with the model using the command line interface (CLI) app or the Python API.

    .. code:: shell

        python
        >>> from mlc_llm import MLCEngine
        >>> engine = MLCEngine(model="./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC",
        ...   model_lib="./dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-cuda.so")
        >>> engine.chat.completions.create(
        ...   messages=[{"role": "user", "content": "hello"}]
        ... )
        ChatCompletionResponse(
          choices=[ChatCompletionResponseChoice(
            message=ChatCompletionMessage(
              content="Hi! How can I assist you today?", role='assistant'
            )
          )],
          ...
        )

.. group-tab:: Metal

    .. code:: shell

        ~/mlc-llm > ls dist/libs
          RedPajama-INCITE-Chat-3B-v1-q4f16_1-metal.so     # ===> the model library (will be -metal_x86_64.dylib for Intel Mac)

        ~/mlc-llm > ls dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC
          mlc-chat-config.json                             # ===> the chat config
          tensor-cache.json                               # ===> the model weight info
          params_shard_0.bin                               # ===> the model weights
          params_shard_1.bin
          ...
          tokenizer.json                                   # ===> the tokenizer files
          tokenizer_config.json

    We can now chat with the model using the command line interface (CLI) app or the Python API.

    .. code:: shell

        python
        >>> from mlc_llm import MLCEngine
        >>> engine = MLCEngine(model="./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC",
        ...   model_lib="./dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-metal.so")
        >>> engine.chat.completions.create(
        ...   messages=[{"role": "user", "content": "hello"}]
        ... )
        ChatCompletionResponse(
          choices=[ChatCompletionResponseChoice(
            message=ChatCompletionMessage(
              content="Hi! How can I assist you today?", role='assistant'
            )
          )],
          ...
        )


.. group-tab:: Vulkan

    .. code:: shell

        ~/mlc-llm > ls dist/libs
          RedPajama-INCITE-Chat-3B-v1-q4f16_1-vulkan.so    # ===> the model library (will be .dll for Windows)

        ~/mlc-llm > ls dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC
          mlc-chat-config.json                             # ===> the chat config
          tensor-cache.json                               # ===> the model weight info
          params_shard_0.bin                               # ===> the model weights
          params_shard_1.bin
          ...
          tokenizer.json                                   # ===> the tokenizer files
          tokenizer_config.json

    We can now chat with the model using the command line interface (CLI) app or the Python API.

    .. code:: shell

        python
        >>> from mlc_llm import MLCEngine
        >>> engine = MLCEngine(model="./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC",
        ...   model_lib="./dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-vulkan.so")
        >>> engine.chat.completions.create(
        ...   messages=[{"role": "user", "content": "hello"}]
        ... )
        ChatCompletionResponse(
          choices=[ChatCompletionResponseChoice(
            message=ChatCompletionMessage(
              content="Hi! How can I assist you today?", role='assistant'
            )
          )],
          ...
        )

.. group-tab:: iOS/iPadOS

    .. code:: shell

        ~/mlc-llm > ls dist/libs
          RedPajama-INCITE-Chat-3B-v1-q4f16_1-iphone.tar   # ===> the model library

        ~/mlc-llm > ls dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC
          mlc-chat-config.json                             # ===> the chat config
          tensor-cache.json                               # ===> the model weight info
          params_shard_0.bin                               # ===> the model weights
          params_shard_1.bin
          ...
          tokenizer.json                                   # ===> the tokenizer files
          tokenizer_config.json

    The model lib ``dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-iphone.tar``
    will be packaged as a static library into the iOS app. Checkout :ref:`deploy-ios` for more details.

.. group-tab:: Android

    .. code:: shell

        ~/mlc-llm > ls dist/libs
          RedPajama-INCITE-Chat-3B-v1-q4f16_1-android.tar  # ===> the model library

        ~/mlc-llm > ls dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC
          mlc-chat-config.json                             # ===> the chat config
          tensor-cache.json                               # ===> the model weight info
          params_shard_0.bin                               # ===> the model weights
          params_shard_1.bin
          ...
          tokenizer.json                                   # ===> the tokenizer files
          tokenizer_config.json

    The model lib ``dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-android.tar``
    will be packaged as a static library into the android app. Checkout :ref:`deploy-android` for more details.

.. group-tab:: WebGPU

    .. code:: shell

        ~/mlc-llm > ls dist/libs
          RedPajama-INCITE-Chat-3B-v1-q4f16_1-webgpu.wasm  # ===> the model library

        ~/mlc-llm > ls dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC
          mlc-chat-config.json                             # ===> the chat config
          tensor-cache.json                               # ===> the model weight info
          params_shard_0.bin                               # ===> the model weights
          params_shard_1.bin
          ...
          tokenizer.json                                   # ===> the tokenizer files
          tokenizer_config.json

    To use this in WebGPU runtime, checkout :ref:`webllm-runtime`.

Compile Commands for More Models

This section lists compile commands for more models that you can try out. Note that this can be easily
generalized to any model variant, as long as mlc-llm supports the architecture.

.. tabs::

text

.. tab:: Model: Llama-2-7B

    Please `request for access <https://huggingface.co/meta-llama>`_ to the Llama-2 weights from Meta first.
    After granted access, first create directory ``dist/models`` and download the model to the directory.
    For example, you can run the following code:

    .. code:: shell

        mkdir -p dist/models && cd dist/models
        git lfs install
        git clone https://huggingface.co/meta-llama/Llama-2-7b-chat-hf
        cd ../..

    Then convert the HF weights into MLC-compatible weights. Note that all platforms
    can share the same compiled/quantized weights.

    .. code:: shell

        mlc_llm convert_weight ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC

    Afterwards, run the following command to generate mlc config and compile the model.

    .. code:: shell

        # Create output directory for the model library compiled
        mkdir dist/libs

    .. tabs::

        .. tab:: Target: CUDA

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \
                    --conv-template llama-2 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \
                    --device cuda -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-cuda.so

        .. tab:: Metal

            For M-chip Mac:

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \
                    --conv-template llama-2 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \
                    --device metal -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-metal.so

            Cross-Compiling for Intel Mac on M-chip Mac:

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \
                    --quantization q4f16_1 --conv-template redpajama_chat \
                    -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \
                    --device metal:x86-64 -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-metal_x86_64.dylib

            For Intel Mac:

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \
                    --conv-template llama-2 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \
                    --device metal -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-metal_x86_64.dylib

        .. tab:: Vulkan

            For Linux:

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \
                    --conv-template llama-2 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \
                    --device vulkan -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-vulkan.so

            For Windows:

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \
                    --conv-template llama-2 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \
                    --device vulkan -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-vulkan.dll

        .. tab:: WebGPU

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \
                    --context-window-size 2048 --conv-template llama-2 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \
                    --device webgpu -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-webgpu.wasm

            .. note::
                To compile for webgpu, you need to build from source when installing ``mlc_llm``. Besides, you also need to follow :ref:`install-web-build`.
                Otherwise, it would run into error

                .. code:: text

                    RuntimeError: Cannot find libraries: wasm_runtime.bc

        .. tab:: iPhone/iPad

            You need a Mac to compile models for it.

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \
                    --conv-template llama-2 --context-window-size 768 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \
                    --device iphone -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-iphone.tar

        .. tab:: Android

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \
                    --conv-template llama-2 --context-window-size 768 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \
                    --device android -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-android.tar

.. tab:: Mistral-7B-Instruct-v0.2

    Note that Mistral uses sliding window attention (SWA). Thus, instead of specifying
    ``context-window-size``, we specify ``sliding-window-size``.

    First create directory ``dist/models`` and download the model to the directory.
    For example, you can run the following code:

    .. code:: shell

        mkdir -p dist/models && cd dist/models
        git lfs install
        git clone https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
        cd ../..

    Then convert the HF weights into MLC-compatible weights. Note that all platforms
    can share the same compiled/quantized weights.

    .. code:: shell

        mlc_llm convert_weight ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \
            -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC

    Afterwards, run the following command to generate mlc config and compile the model.

    .. code:: shell

        # Create output directory for the model library compiled
        mkdir dist/libs

    .. tabs::

        .. tab:: Target: CUDA

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \
                    --conv-template mistral_default -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \
                    --device cuda -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-cuda.so

        .. tab:: Metal

            For M-chip Mac:

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \
                    --conv-template mistral_default -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \
                    --device metal -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-metal.so


            For Intel Mac:

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \
                    --conv-template mistral_default -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \
                    --device metal -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-metal_x86_64.dylib

        .. tab:: Vulkan

            For Linux:

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \
                    --conv-template mistral_default -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \
                    --device vulkan -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-vulkan.so

            For Windows:

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \
                    --conv-template mistral_default -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \
                    --device vulkan -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-vulkan.dll

        .. tab:: WebGPU

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \
                    --prefill-chunk-size 1024 --conv-template mistral_default \
                    -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \
                    --device webgpu -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-webgpu.wasm

            .. note::
                To compile for webgpu, you need to build from source when installing ``mlc_llm``. Besides, you also need to follow :ref:`install-web-build`.
                Otherwise, it would run into error

                .. code:: text

                    RuntimeError: Cannot find libraries: wasm_runtime.bc

            .. note::
                For webgpu, when compiling larger models like ``Llama-2-7B``, you may want to add ``--prefill-chunk-size 1024`` or lower ``--context-window-size`` to decrease memory usage.
                Otherwise, you may run into issues like:

                .. code:: text

                    TypeError: Failed to execute 'createBuffer' on 'GPUDevice': Failed to read the 'size' property from
                    'GPUBufferDescriptor': Value is outside the 'unsigned long long' value range.

        .. tab:: iPhone/iPad

            You need a Mac to compile models for it.

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \
                    --conv-template mistral_default --sliding-window-size 1024 --prefill-chunk-size 128  \
                    -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \
                    --device iphone -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-iphone.tar

        .. tab:: Android

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \
                    --conv-template mistral_default --sliding-window-size 1024 --prefill-chunk-size 128 -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \
                    --device android -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-android.tar

.. tab:: Other models

    First create directory ``dist/models`` and download the model to the directory.
    For example, you can run the following code:

    .. code:: shell

        mkdir -p dist/models && cd dist/models
        git lfs install
        git clone https://huggingface.co/DISTRIBUTOR/HF_MODEL
        cd ../..

    Then convert the HF weights into MLC-compatible weights. Note that all platforms
    can share the same compiled/quantized weights.

    .. code:: shell

        mlc_llm convert_weight ./dist/models/HF_MODEL/ --quantization q4f16_1 -o dist/OUTPUT-MLC

    Afterwards, run the following command to generate mlc config and compile the model.

    .. code:: shell

        # Create output directory for the model library compiled
        mkdir dist/libs

    .. tabs::

        .. tab:: Target: CUDA

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE -o dist/OUTPUT-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device cuda -o dist/libs/OUTPUT-cuda.so

        .. tab:: Metal

            For M-chip Mac:

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE -o dist/OUTPUT-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device metal -o dist/libs/OUTPUT-metal.so


            For Intel Mac:

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE -o dist/OUTPUT-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device metal -o dist/libs/OUTPUT-metal_x86_64.dylib

        .. tab:: Vulkan

            For Linux:

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE -o dist/OUTPUT-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device vulkan -o dist/libs/OUTPUT-vulkan.so

            For Windows:

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE -o dist/OUTPUT-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device vulkan -o dist/libs/OUTPUT-vulkan.dll

        .. tab:: WebGPU

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE -o dist/OUTPUT-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device webgpu -o dist/libs/OUTPUT-webgpu.wasm

            .. note::
                To compile for webgpu, you need to build from source when installing ``mlc_llm``. Besides, you also need to follow :ref:`install-web-build`.
                Otherwise, it would run into error

                .. code:: text

                    RuntimeError: Cannot find libraries: wasm_runtime.bc

            .. note::
                For webgpu, when compiling larger models like ``Llama-2-7B``, you may want to add ``--prefill-chunk-size 1024`` or lower ``--context-window-size`` to decrease memory usage.
                Otherwise, you may run into issues like:

                .. code:: text

                    TypeError: Failed to execute 'createBuffer' on 'GPUDevice': Failed to read the 'size' property from
                    'GPUBufferDescriptor': Value is outside the 'unsigned long long' value range.

        .. tab:: iPhone/iPad

            You need a Mac to compile models for it.

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE \
                    --context-window-size 768 -o dist/OUTPUT-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device iphone -o dist/libs/OUTPUT-iphone.tar

        .. tab:: Android

            .. code:: shell

                # 1. gen_config: generate mlc-chat-config.json and process tokenizers
                mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE \
                    --context-window-size 768 -o dist/OUTPUT-MLC/
                # 2. compile: compile model library with specification in mlc-chat-config.json
                mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device android -o dist/libs/OUTPUT-android.tar

For each model and each backend, the above only provides the most recommended build command (which is the most optimized).
You can also try with different argument values (e.g., different quantization modes, context window size, etc.),
whose build results affect runtime memory requirement, and it is possible that they may not run as
fast and robustly as the provided one when running the model.

.. note::
Uing 3-bit quantization usually can be overly aggressive and only works for limited settings.
If you encounter issues where the compiled model does not perform as expected,
consider utilizing a higher number of bits for quantization (e.g., 4-bit quantization).

If you are interested in distributing the model besides local execution, please checkout :ref:distribute-compiled-models.

.. _compile-command-specification:

Compile Command Specification

As you have seen in the section above, the model compilation is split into three steps: convert weights, generate
mlc-chat-config.json, and compile the model. This section describes the list of options that can be used
during compilation.

Convert Weight
^^^^^^^^^^^^^^^^^

Weight conversion command follows the pattern below:

.. code:: text

text

mlc_llm convert_weight \
    CONFIG \
    --quantization QUANTIZATION_MODE \
    [--model-type MODEL_TYPE] \
    [--device DEVICE] \
    [--source SOURCE] \
    [--source-format SOURCE_FORMAT] \
    --output OUTPUT

Note that CONFIG is a positional argument. Arguments wrapped with [ ] are optional.

--CONFIG It can be one of the following:

text

1. Path to a HuggingFace model directory that contains a ``config.json`` or
                                2. Path to ``config.json`` in HuggingFace format, or
                                3. The name of a pre-defined model architecture.

                                A ``config.json`` file in HuggingFace format defines the model architecture, including the vocabulary
                                size, the number of layers, the hidden size, number of attention heads, etc.
                                Example: https://huggingface.co/codellama/CodeLlama-7b-hf/blob/main/config.json.

                                A HuggingFace directory often contains a ``config.json`` which defines the model architecture,
                                the non-quantized model weights in PyTorch or SafeTensor format, tokenizer configurations,
                                as well as an optional ``generation_config.json`` provides additional default configuration for
                                text generation.
                                Example: https://huggingface.co/codellama/CodeLlama-7b-hf/tree/main.

                                For existing pre-defined model architecture, see ``MODEL_PRESETS``
                                `here <https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/compiler/model/model.py>`_.

--quantization QUANTIZATION_MODE The quantization mode we use to compile.

text

See :ref:`quantization_mode` for more information.
                                Available options are: ``q0f16``, ``q0f32``, ``q3f16_1``, ``q4f16_1``, ``q4f32_1``, and
                                ``q4f16_awq``.

                                We encourage you to use 4-bit quantization, as the text generated by 3-bit
                                quantized models may have bad quality depending on the model.

--model-type MODEL_TYPE Model architecture such as "llama". If not set, it is inferred from config.json.

--device DEVICE The device used to do quantization such as "cuda" or "cuda:0". Will detect from
local available GPUs if not specified.

--source SOURCE The path to original model weight, infer from config if missing.

--source-format SOURCE_FORMAT The format of source model weight, infer from config if missing.

--output OUTPUT The output directory to save the quantized model weight.
Will create params_shard_*.bin and tensor-cache.json in this directory.

Generate MLC Chat Config
^^^^^^^^^^^^^^^^^^^^^^^^^^^

In order to compile a model, we first need to generate the mlc-chat-config.json. This file contains specifications
like context-window-size and sliding-window-size, among others that can alter the model compiled. We also process
tokenizers in this step.

Config generation command follows the pattern below:

.. code:: text

text

mlc_llm gen_config \
    CONFIG \
    --quantization QUANTIZATION_MODE \
    [--model-type MODEL_TYPE] \
    --conv-template CONV_TEMPLATE \
    [--context-window-size CONTEXT_WINDOW_SIZE] \
    [--sliding-window-size SLIDING_WINDOW_SIZE] \
    [--prefill-chunk-size PREFILL_CHUNK_SIZE] \
    [--tensor-parallel-shard TENSOR_PARALLEL_SHARDS] \
    --output OUTPUT

Note that CONFIG is a positional argument. Arguments wrapped with [ ] are optional.

--CONFIG It can be one of the following:

text

1. Path to a HuggingFace model directory that contains a ``config.json`` or
                                            2. Path to ``config.json`` in HuggingFace format, or
                                            3. The name of a pre-defined model architecture.

                                            A ``config.json`` file in HuggingFace format defines the model architecture, including the vocabulary
                                            size, the number of layers, the hidden size, number of attention heads, etc.
                                            Example: https://huggingface.co/codellama/CodeLlama-7b-hf/blob/main/config.json.

                                            A HuggingFace directory often contains a ``config.json`` which defines the model architecture,
                                            the non-quantized model weights in PyTorch or SafeTensor format, tokenizer configurations,
                                            as well as an optional ``generation_config.json`` provides additional default configuration for
                                            text generation.
                                            Example: https://huggingface.co/codellama/CodeLlama-7b-hf/tree/main.

                                            For existing pre-defined model architecture, see ``MODEL_PRESETS``
                                            `here <https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/compiler/model/model.py>`_.

--quantization QUANTIZATION_MODE The quantization mode we use to compile.

text

See :ref:`quantization_mode` for more information.
                                            Available options are: ``q0f16``, ``q0f32``, ``q3f16_1``, ``q4f16_1``, ``q4f32_1``, and
                                            ``q4f16_awq``.

                                            We encourage you to use 4-bit quantization, as the text generated by 3-bit
                                            quantized models may have bad quality depending on the model.

--model-type MODEL_TYPE Model architecture such as "llama". If not set, it is inferred from config.json.

--conv-template CONV_TEMPLATE Conversation template. It depends on how the model is tuned. Use "LM" for vanilla base model
For existing pre-defined templates, see CONV_TEMPLATES
here <https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/model/model.py>_.

--context-window-size CONTEXT_WINDOW_SIZE Option to provide the maximum sequence length supported by the model.
This is usually explicitly shown as context length or context window in the model card.
If this option is not set explicitly, by default,
it will be determined by context_window_size or max_position_embeddings in config.json,
and the latter is usually inaccurate for some models.

--sliding-window-size SLIDING_WINDOW (Experimental) The sliding window size in sliding window attention (SWA).
This optional field overrides the sliding_window in config.json for
those models that use SWA. Currently only useful when compiling mistral-based models.
This flag subjects to future refactoring.

--prefill-chunk-size PREFILL_CHUNK_SIZE (Experimental) The chunk size during prefilling. By default,
the chunk size is the same as context_window_size or sliding_window_size.
This flag subjects to future refactoring.

--tensor-parallel-shard TENSOR_PARALLEL_SHARDS Number of shards to split the model into in tensor parallelism multi-gpu inference.

--output OUTPUT The output directory for generated configurations, including mlc-chat-config.json and tokenizer configuration.

Compile Model Library
^^^^^^^^^^^^^^^^^^^^^^^^

After generating mlc-chat-config.json, we can compile the model into a model library (files ending in .so, .tar, etc. that contains
the inference logic of a model).

Model compilation command follows the pattern below:

.. code:: text

text

mlc_llm compile \
    MODEL \
    [--quantization QUANTIZATION_MODE] \
    [--model-type MODEL_TYPE] \
    [--device DEVICE] \
    [--host HOST] \
    [--opt OPT] \
    [--system-lib-prefix SYSTEM_LIB_PREFIX] \
    --output OUTPUT \
    [--overrides OVERRIDES]

Note that MODEL is a positional argument. Arguments wrapped with [ ] are optional.

--MODEL A path to mlc-chat-config.json, or an MLC model directory that contains mlc-chat-config.json.

--quantization QUANTIZATION_MODE The quantization mode we use to compile. If unprovided, will infer from MODEL.

text

See :ref:`quantization_mode` for more information.
                                        Available options are: ``q0f16``, ``q0f32``, ``q3f16_1``, ``q4f16_1``, ``q4f32_1``, and
                                        ``q4f16_awq``.

                                        We encourage you to use 4-bit quantization, as the text generated by 3-bit
                                        quantized models may have bad quality depending on the model.

--model-type MODEL_TYPE Model architecture such as "llama". If not set, it is inferred from mlc-chat-config.json.

--device DEVICE The GPU device to compile the model to. If not set, it is inferred from GPUs available locally.

--host HOST The host LLVM triple to compile the model to. If not set, it is inferred from the local CPU and OS.
Examples of the LLVM triple:

text

1) iPhones: arm64-apple-ios;
                                        2) ARM64 Android phones: aarch64-linux-android;
                                        3) WebAssembly: wasm32-unknown-unknown-wasm;
                                        4) Windows: x86_64-pc-windows-msvc;
                                        5) ARM macOS: arm64-apple-darwin.

--opt OPT Optimization flags. MLC LLM maintains a predefined set of optimization flags,
denoted as O0, O1, O2, O3, where O0 means no optimization, O2
means majority of them, and O3 represents extreme optimization that could
potentially break the system.

text

Meanwhile, optimization flags could be explicitly specified via details knobs, e.g.
                                        ``--opt="cutlass_attn=1;cutlass_norm=0;cublas_gemm=0;cudagraph=0"``.

--system-lib-prefix SYSTEM_LIB_PREFIX Adding a prefix to all symbols exported. Similar to objcopy --prefix-symbols.
This is useful when compiling multiple models into a single library to avoid symbol
conflicts. Different from objcopy, this takes no effect for shared library.

--output OUTPUT The path to the output file. The suffix determines if the output file is a shared library or
objects. Available suffixes:

text

1) Linux: .so (shared), .tar (objects);
                                        2) macOS: .dylib (shared), .tar (objects);
                                        3) Windows: .dll (shared), .tar (objects);
                                        4) Android, iOS: .tar (objects);
                                        5) Web: .wasm (web assembly).

--overrides OVERRIDES Model configuration override. Configurations to override mlc-chat-config.json. Supports
context_window_size, prefill_chunk_size, sliding_window, max_batch_size and
tensor_parallel_shards. Meanwhile, model config could be explicitly specified via details
knobs, e.g. --overrides "context_window_size=1024;prefill_chunk_size=128".

File: docs/compilation/configure_quantization.rst

Configure Quantization

Quantization Algorithm

The default quantization algorithm used in MLC-LLM is grouping quantization method discussed in the papers The case for 4-bit precision: k-bit Inference Scaling Laws <https://arxiv.org/abs/2212.09720>__ and LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models <https://arxiv.org/abs/2206.09557>__.

.. _quantization_mode:

Quantization Mode

In MLC-LLM we use a short code that indicates the quantization mode to use. MLC-LLM supports both
weight-only quantization and weight-activation quantization.

For the weight-only quantization, he format of the code is qAfB(_id), where A represents the number
of bits for storing weights and B represents the number of bits for storing activations.
The _id is an integer identifier to distinguish different quantization algorithms (e.g. symmetric, non-symmetric, AWQ, etc).

Currently, available options are: q0f16, q0f32, q3f16_1, q4f16_1, q4f32_1, and q4f16_awq (not stable).

For the weight-activation quantization, currently MLC-LLM supports FP8 quantization on CUDA.
The available options are: e4m3_e4m3_f16 and e5m2_e5m2_f16. In these modes, both weights and activations are quantized to FP8 format.
The output of each layer is in higher precision (FP16) and then requantized to FP8.

.. _calibration:

Calibration

For e4m3_e4m3_f16 quantization, we need to calibrate the quantization parameters for the activations.
The calibration process is done by running the following command:

Compile the calibration model
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We use the same compilation workflow to compile the model in calibration mode.
The only difference is that we need to specify the quantization mode as e4m3_e4m3_f16_calibrate.

.. code-block:: bash

text

mlc_llm gen_config \
    <model-path> \
    --quantization e4m3_e4m3_f16_max_calibrate \
    --output <output-path>

mlc_llm convert_weights \
    <model-path> \
    --quantization e4m3_e4m3_f16_max_calibrate \
    --output <output-path>

mlc_llm compile \
    <config-path> \
    --output <output-path>

Run the calibration model
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We will run the calibration model on the dataset such as ShareGPT to collect the statistics of the
activations. The calibration model will updates the quantization parameters in the weights file
in-place. We turn off the cuda graph as it is not yet supported in the calibration process.

.. code-block:: bash

mlc_llm calibrate

--model-lib
--dataset
--num-calibration-samples
--opt "cudagraph=0"
--output

Compile the quantized model for inference.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

After the calibration process, we can compile the model for inference. In this step, we only need
to generate the configuration file using the desired quantization format and compile the model.
Weights are already quantized and calibrated in the previous steps and do not need to be converted again.

.. code-block:: bash

text

mlc_llm gen_config \
    <model-path> \
    --quantization e4m3_e4m3_f16 \
    --output <output-path>
mlc_llm compile \
    <config-path> \
    --output <output-path>

File: docs/compilation/convert_weights.rst

.. _convert-weights-via-MLC:

Convert Model Weights

To run a model with MLC LLM,
we need to convert model weights into MLC format (e.g. RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC <https://huggingface.co/mlc-ai/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/tree/main>_.)
This page walks us through the process of adding a model variant with mlc_llm convert_weight, which
takes a huggingface model as input and converts/quantizes into MLC-compatible weights.

Specifically, we add RedPjama-INCITE-Instruct-3B-v1, while MLC already
provides a model library for RedPjama-INCITE-Chat-3B-v1, which we can reuse.

This can be extended to, e.g.:

Add OpenHermes-Mistral when MLC already supports Mistral
Add Llama-2-uncensored when MLC already supports Llama-2

.. note::
Before you proceed, make sure you followed :ref:install-tvm, a required
backend to compile models with MLC LLM.

text

Please also follow the instructions in :ref:`deploy-cli` / :ref:`deploy-python-engine` to obtain
the CLI app / Python API that can be used to chat with the compiled model.

.. contents:: Table of Contents
:depth: 1
:local:

.. _verify_installation_for_compile:

Verify installation

Step 1. Verify mlc_llm

.. code:: bash

text

$ mlc_llm --help
# You should see help information with this line
usage: MLC LLM Command Line Interface. [-h] {compile,convert_weight,gen_config}

.. note::
If it runs into error command not found: mlc_llm, try python -m mlc_llm --help.

Step 2. Verify TVM

To compile models, you also need to follow :ref:install-tvm.
Here we verify tvm quickly with command line (for full verification, see :ref:tvm-validate):

.. code:: bash

text

$ python -c "import tvm; print(tvm.__file__)"
/some-path/lib/python3.13/site-packages/tvm/__init__.py

Clone from HF and convert_weight

You can be under the mlc-llm repo, or your own working directory. Note that all platforms
can share the same compiled/quantized weights. See :ref:compile-command-specification
for specification of convert_weight.

.. code:: shell

text

# Create directory
mkdir -p dist/models && cd dist/models
# Clone HF weights
git lfs install
git clone https://huggingface.co/togethercomputer/RedPajama-INCITE-Instruct-3B-v1
cd ../..
# Convert weight
mlc_llm convert_weight ./dist/models/RedPajama-INCITE-Instruct-3B-v1/ \
    --quantization q4f16_1 \
    -o dist/RedPajama-INCITE-Instruct-3B-v1-q4f16_1-MLC

.. _generate_mlc_chat_config:

Generate MLC Chat Config

Use mlc_llm gen_config to generate mlc-chat-config.json and process tokenizers.
See :ref:compile-command-specification for specification of gen_config.

.. code:: shell

text

mlc_llm gen_config ./dist/models/RedPajama-INCITE-Instruct-3B-v1/ \
    --quantization q4f16_1 --conv-template redpajama_chat \
    -o dist/RedPajama-INCITE-Instruct-3B-v1-q4f16_1-MLC/

.. note::
The file mlc-chat-config.json is crucial in both model compilation
and runtime chatting. Here we only care about the latter case.

text

You can **optionally** customize
``dist/RedPajama-INCITE-Instruct-3B-v1-q4f16_1-MLC/mlc-chat-config.json`` (checkout :ref:`configure-mlc-chat-json` for more detailed instructions).
You can also simply use the default configuration.

`conversation_template <https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/conversation_template>`__
directory contains a full list of conversation templates that MLC provides. If the model you are adding
requires a new conversation template, you would need to add your own.
Follow `this PR <https://github.com/mlc-ai/mlc-llm/pull/2163>`__ as an example. However,
adding your own template would require you :ref:`build mlc_llm from source <mlcchat_build_from_source>` in order for it
to be recognized by the runtime.

By now, you should have the following files.

.. code:: shell

text

~/mlc-llm > ls dist/RedPajama-INCITE-Instruct-3B-v1-q4f16_1-MLC
    mlc-chat-config.json                             # ===> the chat config
    tensor-cache.json                               # ===> the model weight info
    params_shard_0.bin                               # ===> the model weights
    params_shard_1.bin
    ...
    tokenizer.json                                   # ===> the tokenizer files
    tokenizer_config.json

.. _distribute-compiled-models:

(Optional) 3. Upload weights to HF

Optionally, you can upload what we have to huggingface.

.. code:: shell

text

# First, please create a repository on Hugging Face.
# With the repository created, run
git lfs install
git clone https://huggingface.co/my-huggingface-account/my-redpajama3b-weight-huggingface-repo
cd my-redpajama3b-weight-huggingface-repo
cp path/to/mlc-llm/dist/RedPajama-INCITE-Instruct-3B-v1-q4f16_1-MLC/* .
git add . && git commit -m "Add redpajama-3b instruct model weights"
git push origin main

This would result in something like RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC <https://huggingface.co/mlc-ai/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/tree/main>_, but
for Instruct instead of Chat.

Good job, you have successfully distributed the model you compiled.
Next, we will talk about how we can consume the model weights in applications.

Download the Distributed Models

You can now use the existing mlc tools such as chat/serve/package with the converted weights.

.. code:: shell

text

mlc_llm chat HF://my-huggingface-account/my-redpajama3b-weight-huggingface-repo

File: docs/compilation/define_new_models.rst

Define New Model Architectures

This page guides you how to add a new model architecture in MLC.

This notebook (runnable in Colab) should contain all necessary information to add a model in
MLC LLM:
https://github.com/mlc-ai/notebooks/blob/main/mlc-llm/tutorial_add_new_model_architecture_in_tvm_nn_module.ipynb

In the notebook, we leverage tvm.nn.module to define a model in MLC LLM. We also use JIT
(just-in-time compilation) to debug the implementation.

You can also refer to the PRs below on specific examples of adding a model architecture in MLC LLM:

GPTNeoX PR <https://github.com/mlc-ai/mlc-llm/pull/1408>_
GPT-2 PR <https://github.com/mlc-ai/mlc-llm/pull/1314>_
Mistral PR <https://github.com/mlc-ai/mlc-llm/pull/1230>_

.. note::

text

When adding a model variant that has
its architecture already supported in mlc-llm , you **only need to convert weights**
(e.g. adding ``CodeLlama`` when MLC supports ``llama-2``; adding ``OpenHermes Mistral``
when MLC supports ``mistral``). On the other hand, a new model architecture
(or inference logic) requires more work (following the tutorial above).

File: README.md Copy Chapter

About Copy Chapter

Get Started Copy Chapter

Citation Copy Chapter

File: android/MLCChat/README.md Copy Chapter

MLC-LLM Android

File: android/MLCEngineExample/README.md Copy Chapter

MLC-LLM Android

File: android/README.md Copy Chapter

MLC-LLM Android

File: docs/community/faq.rst Copy Chapter

Frequently Asked Questions

File: docs/community/guideline.rst Copy Chapter

Community Guideline

Participate in Community Discussions Copy Chapter

Contribute to MLC-LLM Copy Chapter

General Development Process Copy Chapter

File: docs/compilation/compile_models.rst Copy Chapter

Compile Model Libraries

Compile Commands for More Models Copy Chapter

Compile Command Specification Copy Chapter

File: docs/compilation/configure_quantization.rst Copy Chapter

Configure Quantization

Quantization Algorithm Copy Chapter

Quantization Mode Copy Chapter

Calibration Copy Chapter

File: docs/compilation/convert_weights.rst Copy Chapter

Convert Model Weights

(Optional) 3. Upload weights to HF Copy Chapter

Download the Distributed Models Copy Chapter

File: docs/compilation/define_new_models.rst Copy Chapter

Define New Model Architectures

Explore Related Technologies & Topics

File: README.md

About

Get Started

Citation

File: android/MLCChat/README.md

File: android/MLCEngineExample/README.md

File: android/README.md

File: docs/community/faq.rst

File: docs/community/guideline.rst

Participate in Community Discussions

Contribute to MLC-LLM

General Development Process

File: docs/compilation/compile_models.rst

Compile Commands for More Models

Compile Command Specification

File: docs/compilation/configure_quantization.rst

Quantization Algorithm

Quantization Mode

Calibration

File: docs/compilation/convert_weights.rst

(Optional) 3. Upload weights to HF

Download the Distributed Models

File: docs/compilation/define_new_models.rst