## File: README.md ## About MLC LLM is a machine learning compiler and high-performance deployment engine for large language models. The mission of this project is to enable everyone to develop, optimize, and deploy AI models natively on everyone's platforms.  MLC LLM compiles and runs code on MLCEngine -- a unified high-performance LLM inference engine across the above platforms. MLCEngine provides OpenAI-compatible API available through REST server, python, javascript, iOS, Android, all backed by the same engine and compiler that we keep improving with the community. ## Get Started Please visit our [documentation](https://llm.mlc.ai/docs/) to get started with MLC LLM. - [Installation](https://llm.mlc.ai/docs/install/mlc_llm) - [Quick start](https://llm.mlc.ai/docs/get_started/quick_start) - [Introduction](https://llm.mlc.ai/docs/get_started/introduction) ## Citation Please consider citing our project if you find it useful: ```bibtex @software{mlc-llm, author = {{MLC team}}, title = {{MLC-LLM}}, url = {https://github.com/mlc-ai/mlc-llm}, year = {2023-2025} } ``` The underlying techniques of MLC LLM include: References (Click to expand) ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ## File: android/MLCChat/README.md # MLC-LLM Android Checkout [Documentation page](https://llm.mlc.ai/docs/deploy/android.html) for more information. - run `mlc_llm package` - open this `MLCChat/` folder as a project in Android Studio --- ## File: android/MLCEngineExample/README.md # MLC-LLM Android Checkout [Documentation page](https://llm.mlc.ai/docs/deploy/android.html) for more information. - run `mlc_llm package` - open this `MLCEngineExample/` folder as a project in Android Studio --- ## File: android/README.md # MLC-LLM Android [Documentation page](https://llm.mlc.ai/docs/deploy/android.html) --- ## File: docs/community/faq.rst .. _FAQ: Frequently Asked Questions ========================== This is a list of Frequently Asked Questions (FAQ) about the MLC-LLM. Feel free to suggest new entries! ... How can I customize the temperature, and repetition penalty of models? Please check our :ref:`configure-mlc-chat-json` tutorial. ... What's the quantization algorithm MLC-LLM using? Please check our :doc:`/compilation/configure_quantization` tutorial. ... Why do I encounter an error ``free(): invalid pointer, Aborted (core dumped)`` at the end of model compilation? This happens if you compiled TVM from source and didn't hide LLVM symbols in cmake configurations. Please follow our instructions in :ref:`Building TVM from Source ` tutorial to compile TVM which hides LLVM symbols, or use our pre-built MLC-LLM :doc:`pip wheels <../install/mlc_llm>`. --- ## File: docs/community/guideline.rst .. _community_guide: Community Guideline =================== .. contents:: :depth: 2 :local: Welcome to the MLC-LLM community! Just like you, all of us are in awe of the immense power of large language models. Our goal for MLC-LLM is to foster a project that is driven by an open-source community, working together to democratize this technology and make it accessible across various devices. We are thrilled to have you as part of our community and eagerly anticipate your valuable contributions. .. _community_discussion: Participate in Community Discussions ------------------------------------ We encourage open discussions. If you encounter a bug or have a feature request, please file an issue in MLC-LLM's GitHub `issue tracker `__. You are encouraged to tag the issue with labels such as "bug," "feature request," or "iOS" so that the relevant developers can quickly notice your concern. Additionally, we have set up a `discord server `__ for online discussions. While we encourage participation in the Discord server, we also recommend creating a GitHub issue even if the topic has been discussed there. This ensures that the discussion is archived and searchable for future reference. Before submitting an issue, we kindly ask you to check our :doc:`/community/faq` to see if your question has already been answered. .. _contribute-to-mlc-llm: Contribute to MLC-LLM --------------------- .. _fork-and-create-pull-requests: Fork and Create Pull Requests ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Ready to contribute to MLC-LLM? Awesome! We are excited to see you are ready to contribute your code. The standard way to make changes to MLC-LLM code base is through creating a `pull-request `__, and we will review your code and merge it to the code base when it is ready. The first step to becoming a developer is to `fork `__ the repository to your own github account, you will notice a repository under ``https://github.com/username/mlc-llm`` where ``username`` is your github user name. You can clone your fork to your local machine and commit changes, or edit the contents of your fork (in the case you are just fixing typos) on GitHub directly. Once your update is complete, you can click the ``contribute`` button and open a pull request to the main repository. .. _contribute-new-models: Contribute New Models to MLC-LLM ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * If you have compiled a model using our :doc:`/compilation/compile_models` tutorial for an existing model architecture, please upload your models to the internet (e.g., Hugging Face) by following :ref:`distribute-compiled-models` tutorial. * If you add a new model variant to MLC-LLM by following our :doc:`/compilation/define_new_models` tutorial. Please create a pull request to add your model architecture (currently model architectures are placed under `relax_models `__ folder). .. _coding-styles: Coding Styles ^^^^^^^^^^^^^ For python codes, we generally follow the `PEP8 style guide `__. The python comments follow `NumPy style `__ python docstrings. To make things easy, you can use `black `__ to automatically format your python code. .. code:: bash pip install black black your_python_file.py For C++ codes, we generally follow the `Google C++ style guide `__. The C++ comments should be `Doxygen compatible `__. Fo your convenience, you can use `clang-format `__ to automatically format your C++ code. .. code:: bash clang-format -i your_cpp_file.cpp .. _general-development-process: General Development Process --------------------------- Everyone in the community is welcome to send patches, documents, and propose new directions to the project. The key guideline here is to enable everyone in the community to get involved and participate in the decision and development. We encourage public discussion in different channels, so that everyone in the community can participate and get informed in developments. Code reviews are one of the key ways to ensure the quality of the code. High-quality code reviews prevent technical debt for long-term and are crucial to the success of the project. A pull request needs to be reviewed before it gets merged. A committer who has the expertise of the corresponding area would moderate the pull request and merge the code when it is ready. The corresponding committer could request multiple reviewers who are familiar with the area of the code. We encourage contributors to request code reviews themselves and help review each other's code -- remember everyone is volunteering their time to the community, high-quality code review itself costs as much as the actual code contribution, you could get your code quickly reviewed if you do others the same favor. The community should strive to reach a consensus on technical decisions through discussion. We expect committers to moderate technical discussions in a diplomatic way, and provide suggestions with clear technical reasoning when necessary. .. _roles-committers: Committers ^^^^^^^^^^ Committers are individuals who are granted with write access to the project. A committer is usually responsible for a certain area or several areas of the code where they oversee the code review process. The area of contribution can take all forms, including code contributions and code reviews, documents, education, and outreach. The review of pull requests will be assigned to the committers who recently contribute to the area this PR belongs to. Committers are essential for a high quality and healthy project. The community actively looks for new committers from contributors. Each existing committer can nominate new committers to MLC projects. .. _roles-contributors: Contributors ^^^^^^^^^^^^ We also welcome contributors if you are not ready to be a committer yet. Everyone who contributes to the project (in the form of code, bugfix, documentation, tutorials, etc) is a contributor. We maintain a `page `__ to acknowledge contributors, please let us know if you contribute to the project and if your name is not included in the list. --- ## File: docs/compilation/compile_models.rst .. _compile-model-libraries: Compile Model Libraries ======================= To run a model with MLC LLM in any platform, we need: 1. **Model weights** converted to MLC format (e.g. `RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC `__.) 2. **Model library** that comprises the inference logic This page describes how to compile a model library with MLC LLM. Model compilation optimizes the model inference for a given platform, allowing users bring their own new model architecture, use different quantization modes, and customize the overall model optimization flow. Notably, in many cases you do not need to explicit call compile. - If you are using the Python API, you can skip specifying ``model_lib`` and the system will JIT compile the library. - If you are building iOS/android package, checkout :ref:`package-libraries-and-weights`, which provides a simpler high-level command that leverages the compile behind the scheme. This page is still helpful to understand the compilation flow behind the scheme, or be used to explicit create model libraries. We compile ``RedPajama-INCITE-Chat-3B-v1`` with ``q4f16_1`` as an example for all platforms. .. note:: Before you proceed, make sure you followed :ref:`install-tvm`, a required backend to compile models with MLC LLM. Please also follow the instructions in :ref:`deploy-cli` / :ref:`deploy-python-engine` to obtain the CLI app / Python API that can be used to chat with the compiled model. .. contents:: Table of Contents :depth: 1 :local: 1. Verify Installation ---------------------- **Step 1. Verify mlc_llm** We use the python package ``mlc_llm`` to compile models. This can be installed by following :ref:`install-mlc-packages`, either by building from source, or by installing the prebuilt package. Verify ``mlc_llm`` installation in command line via: .. code:: bash $ mlc_llm --help # You should see help information with this line usage: MLC LLM Command Line Interface. [-h] {compile,convert_weight,gen_config} .. note:: If it runs into error ``command not found: mlc_llm``, try ``python -m mlc_llm --help``. **Step 2. Verify TVM** To compile models, you also need to follow :ref:`install-tvm`. Here we verify ``tvm`` quickly with command line (for full verification, see :ref:`tvm-validate`): .. code:: bash $ python -c "import tvm; print(tvm.__file__)" /some-path/lib/python3.13/site-packages/tvm/__init__.py 1. Clone from HF and convert_weight ----------------------------------- This replicates :ref:`convert-weights-via-MLC`, see that page for more details. You can be under the mlc-llm repo, or your own working directory. Note that all platforms can share the same compiled/quantized weights. .. code:: shell # Create directory mkdir -p dist/models && cd dist/models # Clone HF weights git lfs install git clone https://huggingface.co/togethercomputer/RedPajama-INCITE-Chat-3B-v1 cd ../.. # Convert weight mlc_llm convert_weight ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \ --quantization q4f16_1 \ -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC 2. Generate mlc-chat-config and compile --------------------------------------- A model library is specified by: - The model architecture (e.g. ``llama-2``, ``gpt-neox``) - Quantization (e.g. ``q4f16_1``, ``q0f32``) - Metadata (e.g. ``context_window_size``, ``sliding_window_size``, ``prefill-chunk-size``), which affects memory planning - Platform (e.g. ``cuda``, ``webgpu``, ``iOS``) All these knobs are specified in ``mlc-chat-config.json`` generated by ``gen_config``. .. code:: shell # Create output directory for the model library compiled mkdir dist/libs .. tabs:: .. group-tab:: Linux - CUDA .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \ --quantization q4f16_1 --conv-template redpajama_chat \ -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \ --device cuda -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-cuda.so .. group-tab:: Metal For M-chip Mac: .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \ --quantization q4f16_1 --conv-template redpajama_chat \ -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \ --device metal -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-metal.so Cross-Compiling for Intel Mac on M-chip Mac: .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \ --quantization q4f16_1 --conv-template redpajama_chat \ -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \ --device metal:x86-64 -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-metal_x86_64.dylib For Intel Mac: .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \ --quantization q4f16_1 --conv-template redpajama_chat \ -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \ --device metal -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-metal_x86_64.dylib .. group-tab:: Vulkan For Linux: .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \ --quantization q4f16_1 --conv-template redpajama_chat \ -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \ --device vulkan -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-vulkan.so For Windows: .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \ --quantization q4f16_1 --conv-template redpajama_chat \ -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \ --device vulkan -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-vulkan.dll .. group-tab:: iOS/iPadOS You need a Mac to compile models for it. .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ --quantization q4f16_1 \ --conv-template redpajama_chat --context-window-size 768 \ -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \ --device iphone -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-iphone.tar .. note:: If it runs into error .. code:: text Compilation error: xcrun: error: unable to find utility "metal", not a developer tool or in PATH xcrun: error: unable to find utility "metallib", not a developer tool or in PATH , please check and make sure you have Command Line Tools for Xcode installed correctly. You can use ``xcrun metal`` to validate: when it prints ``metal: error: no input files``, it means the Command Line Tools for Xcode is installed and can be found, and you can proceed with the model compiling. .. group-tab:: Android .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ --quantization q4f16_1 \ --conv-template redpajama_chat --context-window-size 768 \ -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \ --device android -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-android.tar .. group-tab:: WebGPU .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \ --quantization q4f16_1 --conv-template redpajama_chat \ -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \ --device webgpu -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-webgpu.wasm .. note:: To compile for webgpu, you need to build from source when installing ``mlc_llm``. Besides, you also need to follow :ref:`install-web-build`. Otherwise, it would run into error .. code:: text RuntimeError: Cannot find libraries: wasm_runtime.bc .. note:: For webgpu, when compiling larger models like ``Llama-2-7B``, you may want to add ``--prefill-chunk-size 1024`` or lower ``--context-window-size`` to decrease memory usage. Otherwise, you may run into issues like: .. code:: text TypeError: Failed to execute 'createBuffer' on 'GPUDevice': Failed to read the 'size' property from 'GPUBufferDescriptor': Value is outside the 'unsigned long long' value range. .. note:: For the ``conv-template``, `conversation_template.py `__ contains a full list of conversation templates that MLC provides. If the model you are adding requires a new conversation template, you would need to add your own. Follow `this PR `__ as an example. However, adding your own template would require you :ref:`build mlc_llm from source ` in order for it to be recognized by the runtime. For more details, please see :ref:`configure-mlc-chat-json`. 3. Verify output and chat ------------------------- By executing the compile command above, we generate the model weights, model lib, and a chat config. We can check the output with the commands below: .. tabs:: .. group-tab:: Linux - CUDA .. code:: shell ~/mlc-llm > ls dist/libs RedPajama-INCITE-Chat-3B-v1-q4f16_1-cuda.so # ===> the model library ~/mlc-llm > ls dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC mlc-chat-config.json # ===> the chat config tensor-cache.json # ===> the model weight info params_shard_0.bin # ===> the model weights params_shard_1.bin ... tokenizer.json # ===> the tokenizer files tokenizer_config.json We can now chat with the model using the command line interface (CLI) app or the Python API. .. code:: shell python >>> from mlc_llm import MLCEngine >>> engine = MLCEngine(model="./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC", ... model_lib="./dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-cuda.so") >>> engine.chat.completions.create( ... messages=[{"role": "user", "content": "hello"}] ... ) ChatCompletionResponse( choices=[ChatCompletionResponseChoice( message=ChatCompletionMessage( content="Hi! How can I assist you today?", role='assistant' ) )], ... ) .. group-tab:: Metal .. code:: shell ~/mlc-llm > ls dist/libs RedPajama-INCITE-Chat-3B-v1-q4f16_1-metal.so # ===> the model library (will be -metal_x86_64.dylib for Intel Mac) ~/mlc-llm > ls dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC mlc-chat-config.json # ===> the chat config tensor-cache.json # ===> the model weight info params_shard_0.bin # ===> the model weights params_shard_1.bin ... tokenizer.json # ===> the tokenizer files tokenizer_config.json We can now chat with the model using the command line interface (CLI) app or the Python API. .. code:: shell python >>> from mlc_llm import MLCEngine >>> engine = MLCEngine(model="./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC", ... model_lib="./dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-metal.so") >>> engine.chat.completions.create( ... messages=[{"role": "user", "content": "hello"}] ... ) ChatCompletionResponse( choices=[ChatCompletionResponseChoice( message=ChatCompletionMessage( content="Hi! How can I assist you today?", role='assistant' ) )], ... ) .. group-tab:: Vulkan .. code:: shell ~/mlc-llm > ls dist/libs RedPajama-INCITE-Chat-3B-v1-q4f16_1-vulkan.so # ===> the model library (will be .dll for Windows) ~/mlc-llm > ls dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC mlc-chat-config.json # ===> the chat config tensor-cache.json # ===> the model weight info params_shard_0.bin # ===> the model weights params_shard_1.bin ... tokenizer.json # ===> the tokenizer files tokenizer_config.json We can now chat with the model using the command line interface (CLI) app or the Python API. .. code:: shell python >>> from mlc_llm import MLCEngine >>> engine = MLCEngine(model="./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC", ... model_lib="./dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-vulkan.so") >>> engine.chat.completions.create( ... messages=[{"role": "user", "content": "hello"}] ... ) ChatCompletionResponse( choices=[ChatCompletionResponseChoice( message=ChatCompletionMessage( content="Hi! How can I assist you today?", role='assistant' ) )], ... ) .. group-tab:: iOS/iPadOS .. code:: shell ~/mlc-llm > ls dist/libs RedPajama-INCITE-Chat-3B-v1-q4f16_1-iphone.tar # ===> the model library ~/mlc-llm > ls dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC mlc-chat-config.json # ===> the chat config tensor-cache.json # ===> the model weight info params_shard_0.bin # ===> the model weights params_shard_1.bin ... tokenizer.json # ===> the tokenizer files tokenizer_config.json The model lib ``dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-iphone.tar`` will be packaged as a static library into the iOS app. Checkout :ref:`deploy-ios` for more details. .. group-tab:: Android .. code:: shell ~/mlc-llm > ls dist/libs RedPajama-INCITE-Chat-3B-v1-q4f16_1-android.tar # ===> the model library ~/mlc-llm > ls dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC mlc-chat-config.json # ===> the chat config tensor-cache.json # ===> the model weight info params_shard_0.bin # ===> the model weights params_shard_1.bin ... tokenizer.json # ===> the tokenizer files tokenizer_config.json The model lib ``dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-android.tar`` will be packaged as a static library into the android app. Checkout :ref:`deploy-android` for more details. .. group-tab:: WebGPU .. code:: shell ~/mlc-llm > ls dist/libs RedPajama-INCITE-Chat-3B-v1-q4f16_1-webgpu.wasm # ===> the model library ~/mlc-llm > ls dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC mlc-chat-config.json # ===> the chat config tensor-cache.json # ===> the model weight info params_shard_0.bin # ===> the model weights params_shard_1.bin ... tokenizer.json # ===> the tokenizer files tokenizer_config.json To use this in WebGPU runtime, checkout :ref:`webllm-runtime`. Compile Commands for More Models -------------------------------- This section lists compile commands for more models that you can try out. Note that this can be easily generalized to any model variant, as long as mlc-llm supports the architecture. .. tabs:: .. tab:: Model: Llama-2-7B Please `request for access `_ to the Llama-2 weights from Meta first. After granted access, first create directory ``dist/models`` and download the model to the directory. For example, you can run the following code: .. code:: shell mkdir -p dist/models && cd dist/models git lfs install git clone https://huggingface.co/meta-llama/Llama-2-7b-chat-hf cd ../.. Then convert the HF weights into MLC-compatible weights. Note that all platforms can share the same compiled/quantized weights. .. code:: shell mlc_llm convert_weight ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC Afterwards, run the following command to generate mlc config and compile the model. .. code:: shell # Create output directory for the model library compiled mkdir dist/libs .. tabs:: .. tab:: Target: CUDA .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \ --conv-template llama-2 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \ --device cuda -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-cuda.so .. tab:: Metal For M-chip Mac: .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \ --conv-template llama-2 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \ --device metal -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-metal.so Cross-Compiling for Intel Mac on M-chip Mac: .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/RedPajama-INCITE-Chat-3B-v1/ \ --quantization q4f16_1 --conv-template redpajama_chat \ -o dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json \ --device metal:x86-64 -o dist/libs/RedPajama-INCITE-Chat-3B-v1-q4f16_1-metal_x86_64.dylib For Intel Mac: .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \ --conv-template llama-2 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \ --device metal -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-metal_x86_64.dylib .. tab:: Vulkan For Linux: .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \ --conv-template llama-2 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \ --device vulkan -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-vulkan.so For Windows: .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \ --conv-template llama-2 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \ --device vulkan -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-vulkan.dll .. tab:: WebGPU .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \ --context-window-size 2048 --conv-template llama-2 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \ --device webgpu -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-webgpu.wasm .. note:: To compile for webgpu, you need to build from source when installing ``mlc_llm``. Besides, you also need to follow :ref:`install-web-build`. Otherwise, it would run into error .. code:: text RuntimeError: Cannot find libraries: wasm_runtime.bc .. tab:: iPhone/iPad You need a Mac to compile models for it. .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \ --conv-template llama-2 --context-window-size 768 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \ --device iphone -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-iphone.tar .. tab:: Android .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/Llama-2-7b-chat-hf/ --quantization q4f16_1 \ --conv-template llama-2 --context-window-size 768 -o dist/Llama-2-7b-chat-hf-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json \ --device android -o dist/libs/Llama-2-7b-chat-hf-q4f16_1-android.tar .. tab:: Mistral-7B-Instruct-v0.2 Note that Mistral uses sliding window attention (SWA). Thus, instead of specifying ``context-window-size``, we specify ``sliding-window-size``. First create directory ``dist/models`` and download the model to the directory. For example, you can run the following code: .. code:: shell mkdir -p dist/models && cd dist/models git lfs install git clone https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 cd ../.. Then convert the HF weights into MLC-compatible weights. Note that all platforms can share the same compiled/quantized weights. .. code:: shell mlc_llm convert_weight ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \ -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC Afterwards, run the following command to generate mlc config and compile the model. .. code:: shell # Create output directory for the model library compiled mkdir dist/libs .. tabs:: .. tab:: Target: CUDA .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \ --conv-template mistral_default -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \ --device cuda -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-cuda.so .. tab:: Metal For M-chip Mac: .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \ --conv-template mistral_default -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \ --device metal -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-metal.so For Intel Mac: .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \ --conv-template mistral_default -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \ --device metal -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-metal_x86_64.dylib .. tab:: Vulkan For Linux: .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \ --conv-template mistral_default -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \ --device vulkan -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-vulkan.so For Windows: .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \ --conv-template mistral_default -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \ --device vulkan -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-vulkan.dll .. tab:: WebGPU .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \ --prefill-chunk-size 1024 --conv-template mistral_default \ -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \ --device webgpu -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-webgpu.wasm .. note:: To compile for webgpu, you need to build from source when installing ``mlc_llm``. Besides, you also need to follow :ref:`install-web-build`. Otherwise, it would run into error .. code:: text RuntimeError: Cannot find libraries: wasm_runtime.bc .. note:: For webgpu, when compiling larger models like ``Llama-2-7B``, you may want to add ``--prefill-chunk-size 1024`` or lower ``--context-window-size`` to decrease memory usage. Otherwise, you may run into issues like: .. code:: text TypeError: Failed to execute 'createBuffer' on 'GPUDevice': Failed to read the 'size' property from 'GPUBufferDescriptor': Value is outside the 'unsigned long long' value range. .. tab:: iPhone/iPad You need a Mac to compile models for it. .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \ --conv-template mistral_default --sliding-window-size 1024 --prefill-chunk-size 128 \ -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \ --device iphone -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-iphone.tar .. tab:: Android .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/Mistral-7B-Instruct-v0.2/ --quantization q4f16_1 \ --conv-template mistral_default --sliding-window-size 1024 --prefill-chunk-size 128 -o dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/Mistral-7B-Instruct-v0.2-q4f16_1-MLC/mlc-chat-config.json \ --device android -o dist/libs/Mistral-7B-Instruct-v0.2-q4f16_1-android.tar .. tab:: Other models First create directory ``dist/models`` and download the model to the directory. For example, you can run the following code: .. code:: shell mkdir -p dist/models && cd dist/models git lfs install git clone https://huggingface.co/DISTRIBUTOR/HF_MODEL cd ../.. Then convert the HF weights into MLC-compatible weights. Note that all platforms can share the same compiled/quantized weights. .. code:: shell mlc_llm convert_weight ./dist/models/HF_MODEL/ --quantization q4f16_1 -o dist/OUTPUT-MLC Afterwards, run the following command to generate mlc config and compile the model. .. code:: shell # Create output directory for the model library compiled mkdir dist/libs .. tabs:: .. tab:: Target: CUDA .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE -o dist/OUTPUT-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device cuda -o dist/libs/OUTPUT-cuda.so .. tab:: Metal For M-chip Mac: .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE -o dist/OUTPUT-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device metal -o dist/libs/OUTPUT-metal.so For Intel Mac: .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE -o dist/OUTPUT-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device metal -o dist/libs/OUTPUT-metal_x86_64.dylib .. tab:: Vulkan For Linux: .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE -o dist/OUTPUT-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device vulkan -o dist/libs/OUTPUT-vulkan.so For Windows: .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE -o dist/OUTPUT-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device vulkan -o dist/libs/OUTPUT-vulkan.dll .. tab:: WebGPU .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE -o dist/OUTPUT-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device webgpu -o dist/libs/OUTPUT-webgpu.wasm .. note:: To compile for webgpu, you need to build from source when installing ``mlc_llm``. Besides, you also need to follow :ref:`install-web-build`. Otherwise, it would run into error .. code:: text RuntimeError: Cannot find libraries: wasm_runtime.bc .. note:: For webgpu, when compiling larger models like ``Llama-2-7B``, you may want to add ``--prefill-chunk-size 1024`` or lower ``--context-window-size`` to decrease memory usage. Otherwise, you may run into issues like: .. code:: text TypeError: Failed to execute 'createBuffer' on 'GPUDevice': Failed to read the 'size' property from 'GPUBufferDescriptor': Value is outside the 'unsigned long long' value range. .. tab:: iPhone/iPad You need a Mac to compile models for it. .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE \ --context-window-size 768 -o dist/OUTPUT-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device iphone -o dist/libs/OUTPUT-iphone.tar .. tab:: Android .. code:: shell # 1. gen_config: generate mlc-chat-config.json and process tokenizers mlc_llm gen_config ./dist/models/HF_MODEL/ --quantization q4f16_1 --conv-template CONV_TEMPLATE \ --context-window-size 768 -o dist/OUTPUT-MLC/ # 2. compile: compile model library with specification in mlc-chat-config.json mlc_llm compile ./dist/OUTPUT-MLC/mlc-chat-config.json --device android -o dist/libs/OUTPUT-android.tar For each model and each backend, the above only provides the most recommended build command (which is the most optimized). You can also try with different argument values (e.g., different quantization modes, context window size, etc.), whose build results affect runtime memory requirement, and it is possible that they may not run as fast and robustly as the provided one when running the model. .. note:: Uing 3-bit quantization usually can be overly aggressive and only works for limited settings. If you encounter issues where the compiled model does not perform as expected, consider utilizing a higher number of bits for quantization (e.g., 4-bit quantization). If you are interested in distributing the model besides local execution, please checkout :ref:`distribute-compiled-models`. .. _compile-command-specification: Compile Command Specification ----------------------------- As you have seen in the section above, the model compilation is split into three steps: convert weights, generate ``mlc-chat-config.json``, and compile the model. This section describes the list of options that can be used during compilation. 1. Convert Weight ^^^^^^^^^^^^^^^^^ Weight conversion command follows the pattern below: .. code:: text mlc_llm convert_weight \ CONFIG \ --quantization QUANTIZATION_MODE \ [--model-type MODEL_TYPE] \ [--device DEVICE] \ [--source SOURCE] \ [--source-format SOURCE_FORMAT] \ --output OUTPUT Note that ``CONFIG`` is a positional argument. Arguments wrapped with ``[ ]`` are optional. --CONFIG It can be one of the following: 1. Path to a HuggingFace model directory that contains a ``config.json`` or 2. Path to ``config.json`` in HuggingFace format, or 3. The name of a pre-defined model architecture. A ``config.json`` file in HuggingFace format defines the model architecture, including the vocabulary size, the number of layers, the hidden size, number of attention heads, etc. Example: https://huggingface.co/codellama/CodeLlama-7b-hf/blob/main/config.json. A HuggingFace directory often contains a ``config.json`` which defines the model architecture, the non-quantized model weights in PyTorch or SafeTensor format, tokenizer configurations, as well as an optional ``generation_config.json`` provides additional default configuration for text generation. Example: https://huggingface.co/codellama/CodeLlama-7b-hf/tree/main. For existing pre-defined model architecture, see ``MODEL_PRESETS`` `here `_. --quantization QUANTIZATION_MODE The quantization mode we use to compile. See :ref:`quantization_mode` for more information. Available options are: ``q0f16``, ``q0f32``, ``q3f16_1``, ``q4f16_1``, ``q4f32_1``, and ``q4f16_awq``. We encourage you to use 4-bit quantization, as the text generated by 3-bit quantized models may have bad quality depending on the model. --model-type MODEL_TYPE Model architecture such as "llama". If not set, it is inferred from ``config.json``. --device DEVICE The device used to do quantization such as "cuda" or "cuda:0". Will detect from local available GPUs if not specified. --source SOURCE The path to original model weight, infer from ``config`` if missing. --source-format SOURCE_FORMAT The format of source model weight, infer from ``config`` if missing. --output OUTPUT The output directory to save the quantized model weight. Will create ``params_shard_*.bin`` and ```tensor-cache.json``` in this directory. 2. Generate MLC Chat Config ^^^^^^^^^^^^^^^^^^^^^^^^^^^ In order to compile a model, we first need to generate the ``mlc-chat-config.json``. This file contains specifications like ``context-window-size`` and ``sliding-window-size``, among others that can alter the model compiled. We also process tokenizers in this step. Config generation command follows the pattern below: .. code:: text mlc_llm gen_config \ CONFIG \ --quantization QUANTIZATION_MODE \ [--model-type MODEL_TYPE] \ --conv-template CONV_TEMPLATE \ [--context-window-size CONTEXT_WINDOW_SIZE] \ [--sliding-window-size SLIDING_WINDOW_SIZE] \ [--prefill-chunk-size PREFILL_CHUNK_SIZE] \ [--tensor-parallel-shard TENSOR_PARALLEL_SHARDS] \ --output OUTPUT Note that ``CONFIG`` is a positional argument. Arguments wrapped with ``[ ]`` are optional. --CONFIG It can be one of the following: 1. Path to a HuggingFace model directory that contains a ``config.json`` or 2. Path to ``config.json`` in HuggingFace format, or 3. The name of a pre-defined model architecture. A ``config.json`` file in HuggingFace format defines the model architecture, including the vocabulary size, the number of layers, the hidden size, number of attention heads, etc. Example: https://huggingface.co/codellama/CodeLlama-7b-hf/blob/main/config.json. A HuggingFace directory often contains a ``config.json`` which defines the model architecture, the non-quantized model weights in PyTorch or SafeTensor format, tokenizer configurations, as well as an optional ``generation_config.json`` provides additional default configuration for text generation. Example: https://huggingface.co/codellama/CodeLlama-7b-hf/tree/main. For existing pre-defined model architecture, see ``MODEL_PRESETS`` `here `_. --quantization QUANTIZATION_MODE The quantization mode we use to compile. See :ref:`quantization_mode` for more information. Available options are: ``q0f16``, ``q0f32``, ``q3f16_1``, ``q4f16_1``, ``q4f32_1``, and ``q4f16_awq``. We encourage you to use 4-bit quantization, as the text generated by 3-bit quantized models may have bad quality depending on the model. --model-type MODEL_TYPE Model architecture such as "llama". If not set, it is inferred from ``config.json``. --conv-template CONV_TEMPLATE Conversation template. It depends on how the model is tuned. Use "LM" for vanilla base model For existing pre-defined templates, see ``CONV_TEMPLATES`` `here `_. --context-window-size CONTEXT_WINDOW_SIZE Option to provide the maximum sequence length supported by the model. This is usually explicitly shown as context length or context window in the model card. If this option is not set explicitly, by default, it will be determined by ``context_window_size`` or ``max_position_embeddings`` in ``config.json``, and the latter is usually inaccurate for some models. --sliding-window-size SLIDING_WINDOW (Experimental) The sliding window size in sliding window attention (SWA). This optional field overrides the ``sliding_window`` in ``config.json`` for those models that use SWA. Currently only useful when compiling mistral-based models. This flag subjects to future refactoring. --prefill-chunk-size PREFILL_CHUNK_SIZE (Experimental) The chunk size during prefilling. By default, the chunk size is the same as ``context_window_size`` or ``sliding_window_size``. This flag subjects to future refactoring. --tensor-parallel-shard TENSOR_PARALLEL_SHARDS Number of shards to split the model into in tensor parallelism multi-gpu inference. --output OUTPUT The output directory for generated configurations, including `mlc-chat-config.json` and tokenizer configuration. 3. Compile Model Library ^^^^^^^^^^^^^^^^^^^^^^^^ After generating ``mlc-chat-config.json``, we can compile the model into a model library (files ending in ``.so``, ``.tar``, etc. that contains the inference logic of a model). Model compilation command follows the pattern below: .. code:: text mlc_llm compile \ MODEL \ [--quantization QUANTIZATION_MODE] \ [--model-type MODEL_TYPE] \ [--device DEVICE] \ [--host HOST] \ [--opt OPT] \ [--system-lib-prefix SYSTEM_LIB_PREFIX] \ --output OUTPUT \ [--overrides OVERRIDES] Note that ``MODEL`` is a positional argument. Arguments wrapped with ``[ ]`` are optional. --MODEL A path to ``mlc-chat-config.json``, or an MLC model directory that contains ``mlc-chat-config.json``. --quantization QUANTIZATION_MODE The quantization mode we use to compile. If unprovided, will infer from ``MODEL``. See :ref:`quantization_mode` for more information. Available options are: ``q0f16``, ``q0f32``, ``q3f16_1``, ``q4f16_1``, ``q4f32_1``, and ``q4f16_awq``. We encourage you to use 4-bit quantization, as the text generated by 3-bit quantized models may have bad quality depending on the model. --model-type MODEL_TYPE Model architecture such as "llama". If not set, it is inferred from ``mlc-chat-config.json``. --device DEVICE The GPU device to compile the model to. If not set, it is inferred from GPUs available locally. --host HOST The host LLVM triple to compile the model to. If not set, it is inferred from the local CPU and OS. Examples of the LLVM triple: 1) iPhones: arm64-apple-ios; 2) ARM64 Android phones: aarch64-linux-android; 3) WebAssembly: wasm32-unknown-unknown-wasm; 4) Windows: x86_64-pc-windows-msvc; 5) ARM macOS: arm64-apple-darwin. --opt OPT Optimization flags. MLC LLM maintains a predefined set of optimization flags, denoted as ``O0``, ``O1``, ``O2``, ``O3``, where ``O0`` means no optimization, ``O2`` means majority of them, and ``O3`` represents extreme optimization that could potentially break the system. Meanwhile, optimization flags could be explicitly specified via details knobs, e.g. ``--opt="cutlass_attn=1;cutlass_norm=0;cublas_gemm=0;cudagraph=0"``. --system-lib-prefix SYSTEM_LIB_PREFIX Adding a prefix to all symbols exported. Similar to ``objcopy --prefix-symbols``. This is useful when compiling multiple models into a single library to avoid symbol conflicts. Different from objcopy, this takes no effect for shared library. --output OUTPUT The path to the output file. The suffix determines if the output file is a shared library or objects. Available suffixes: 1) Linux: .so (shared), .tar (objects); 2) macOS: .dylib (shared), .tar (objects); 3) Windows: .dll (shared), .tar (objects); 4) Android, iOS: .tar (objects); 5) Web: .wasm (web assembly). --overrides OVERRIDES Model configuration override. Configurations to override ``mlc-chat-config.json``. Supports ``context_window_size``, ``prefill_chunk_size``, ``sliding_window``, ``max_batch_size`` and ``tensor_parallel_shards``. Meanwhile, model config could be explicitly specified via details knobs, e.g. ``--overrides "context_window_size=1024;prefill_chunk_size=128"``. --- ## File: docs/compilation/configure_quantization.rst Configure Quantization ====================== Quantization Algorithm ---------------------- The default quantization algorithm used in MLC-LLM is grouping quantization method discussed in the papers `The case for 4-bit precision: k-bit Inference Scaling Laws `__ and `LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models `__. .. _quantization_mode: Quantization Mode ----------------- In MLC-LLM we use a short code that indicates the quantization mode to use. MLC-LLM supports both weight-only quantization and weight-activation quantization. For the weight-only quantization, he format of the code is ``qAfB(_id)``, where ``A`` represents the number of bits for storing weights and ``B`` represents the number of bits for storing activations. The ``_id`` is an integer identifier to distinguish different quantization algorithms (e.g. symmetric, non-symmetric, AWQ, etc). Currently, available options are: ``q0f16``, ``q0f32``, ``q3f16_1``, ``q4f16_1``, ``q4f32_1``, and ``q4f16_awq`` (not stable). For the weight-activation quantization, currently MLC-LLM supports FP8 quantization on CUDA. The available options are: ``e4m3_e4m3_f16`` and ``e5m2_e5m2_f16``. In these modes, both weights and activations are quantized to FP8 format. The output of each layer is in higher precision (FP16) and then requantized to FP8. .. _calibration: Calibration ----------- For ``e4m3_e4m3_f16`` quantization, we need to calibrate the quantization parameters for the activations. The calibration process is done by running the following command: 1. Compile the calibration model ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ We use the same compilation workflow to compile the model in calibration mode. The only difference is that we need to specify the quantization mode as ``e4m3_e4m3_f16_calibrate``. .. code-block:: bash mlc_llm gen_config \ \ --quantization e4m3_e4m3_f16_max_calibrate \ --output mlc_llm convert_weights \ \ --quantization e4m3_e4m3_f16_max_calibrate \ --output mlc_llm compile \ \ --output 2. Run the calibration model ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ We will run the calibration model on the dataset such as ShareGPT to collect the statistics of the activations. The calibration model will updates the quantization parameters in the weights file in-place. We turn off the cuda graph as it is not yet supported in the calibration process. .. code-block:: bash mlc_llm calibrate \ \ --model-lib \ --dataset \ --num-calibration-samples \ --opt "cudagraph=0" --output 3. Compile the quantized model for inference. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ After the calibration process, we can compile the model for inference. In this step, we only need to generate the configuration file using the desired quantization format and compile the model. Weights are already quantized and calibrated in the previous steps and do not need to be converted again. .. code-block:: bash mlc_llm gen_config \ \ --quantization e4m3_e4m3_f16 \ --output mlc_llm compile \ \ --output --- ## File: docs/compilation/convert_weights.rst .. _convert-weights-via-MLC: Convert Model Weights ===================== To run a model with MLC LLM, we need to convert model weights into MLC format (e.g. `RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC `_.) This page walks us through the process of adding a model variant with ``mlc_llm convert_weight``, which takes a huggingface model as input and converts/quantizes into MLC-compatible weights. Specifically, we add RedPjama-INCITE-**Instruct**-3B-v1, while MLC already provides a model library for RedPjama-INCITE-**Chat**-3B-v1, which we can reuse. This can be extended to, e.g.: - Add ``OpenHermes-Mistral`` when MLC already supports Mistral - Add ``Llama-2-uncensored`` when MLC already supports Llama-2 .. note:: Before you proceed, make sure you followed :ref:`install-tvm`, a required backend to compile models with MLC LLM. Please also follow the instructions in :ref:`deploy-cli` / :ref:`deploy-python-engine` to obtain the CLI app / Python API that can be used to chat with the compiled model. .. contents:: Table of Contents :depth: 1 :local: .. _verify_installation_for_compile: 1. Verify installation ---------------------- **Step 1. Verify mlc_llm** We use the python package ``mlc_llm`` to compile models. This can be installed by following :ref:`install-mlc-packages`, either by building from source, or by installing the prebuilt package. Verify ``mlc_llm`` installation in command line via: .. code:: bash $ mlc_llm --help # You should see help information with this line usage: MLC LLM Command Line Interface. [-h] {compile,convert_weight,gen_config} .. note:: If it runs into error ``command not found: mlc_llm``, try ``python -m mlc_llm --help``. **Step 2. Verify TVM** To compile models, you also need to follow :ref:`install-tvm`. Here we verify ``tvm`` quickly with command line (for full verification, see :ref:`tvm-validate`): .. code:: bash $ python -c "import tvm; print(tvm.__file__)" /some-path/lib/python3.13/site-packages/tvm/__init__.py 1. Clone from HF and convert_weight ----------------------------------- You can be under the mlc-llm repo, or your own working directory. Note that all platforms can share the same compiled/quantized weights. See :ref:`compile-command-specification` for specification of ``convert_weight``. .. code:: shell # Create directory mkdir -p dist/models && cd dist/models # Clone HF weights git lfs install git clone https://huggingface.co/togethercomputer/RedPajama-INCITE-Instruct-3B-v1 cd ../.. # Convert weight mlc_llm convert_weight ./dist/models/RedPajama-INCITE-Instruct-3B-v1/ \ --quantization q4f16_1 \ -o dist/RedPajama-INCITE-Instruct-3B-v1-q4f16_1-MLC .. _generate_mlc_chat_config: 2. Generate MLC Chat Config --------------------------- Use ``mlc_llm gen_config`` to generate ``mlc-chat-config.json`` and process tokenizers. See :ref:`compile-command-specification` for specification of ``gen_config``. .. code:: shell mlc_llm gen_config ./dist/models/RedPajama-INCITE-Instruct-3B-v1/ \ --quantization q4f16_1 --conv-template redpajama_chat \ -o dist/RedPajama-INCITE-Instruct-3B-v1-q4f16_1-MLC/ .. note:: The file ``mlc-chat-config.json`` is crucial in both model compilation and runtime chatting. Here we only care about the latter case. You can **optionally** customize ``dist/RedPajama-INCITE-Instruct-3B-v1-q4f16_1-MLC/mlc-chat-config.json`` (checkout :ref:`configure-mlc-chat-json` for more detailed instructions). You can also simply use the default configuration. `conversation_template `__ directory contains a full list of conversation templates that MLC provides. If the model you are adding requires a new conversation template, you would need to add your own. Follow `this PR `__ as an example. However, adding your own template would require you :ref:`build mlc_llm from source ` in order for it to be recognized by the runtime. By now, you should have the following files. .. code:: shell ~/mlc-llm > ls dist/RedPajama-INCITE-Instruct-3B-v1-q4f16_1-MLC mlc-chat-config.json # ===> the chat config tensor-cache.json # ===> the model weight info params_shard_0.bin # ===> the model weights params_shard_1.bin ... tokenizer.json # ===> the tokenizer files tokenizer_config.json .. _distribute-compiled-models: (Optional) 3. Upload weights to HF ---------------------------------- Optionally, you can upload what we have to huggingface. .. code:: shell # First, please create a repository on Hugging Face. # With the repository created, run git lfs install git clone https://huggingface.co/my-huggingface-account/my-redpajama3b-weight-huggingface-repo cd my-redpajama3b-weight-huggingface-repo cp path/to/mlc-llm/dist/RedPajama-INCITE-Instruct-3B-v1-q4f16_1-MLC/* . git add . && git commit -m "Add redpajama-3b instruct model weights" git push origin main This would result in something like `RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC `_, but for **Instruct** instead of **Chat**. Good job, you have successfully distributed the model you compiled. Next, we will talk about how we can consume the model weights in applications. Download the Distributed Models ------------------------------- You can now use the existing mlc tools such as chat/serve/package with the converted weights. .. code:: shell mlc_llm chat HF://my-huggingface-account/my-redpajama3b-weight-huggingface-repo --- ## File: docs/compilation/define_new_models.rst Define New Model Architectures ============================== This page guides you how to add a new model architecture in MLC. This notebook (runnable in Colab) should contain all necessary information to add a model in MLC LLM: https://github.com/mlc-ai/notebooks/blob/main/mlc-llm/tutorial_add_new_model_architecture_in_tvm_nn_module.ipynb In the notebook, we leverage ``tvm.nn.module`` to define a model in MLC LLM. We also use ``JIT`` (just-in-time compilation) to debug the implementation. You can also refer to the PRs below on specific examples of adding a model architecture in MLC LLM: - `GPTNeoX PR `_ - `GPT-2 PR `_ - `Mistral PR `_ .. note:: When adding a model variant that has its architecture already supported in mlc-llm , you **only need to convert weights** (e.g. adding ``CodeLlama`` when MLC supports ``llama-2``; adding ``OpenHermes Mistral`` when MLC supports ``mistral``). On the other hand, a new model architecture (or inference logic) requires more work (following the tutorial above).