Repository: alibaba/MNN

Stars: 14933

CLAUDE.md Copy Chapter

MNN Project Instructions

MNN is a lightweight deep learning inference engine (not a training framework), targeting mobile and server platforms. Supports CNN / Transformer / LLM / Diffusion models. Code must prioritize performance and binary size.

Restricted Access Copy Chapter

The following directories contain internal proprietary code. Do NOT read, modify, or reference any files within:

- schema/private/

- source/internal/

Architecture Overview Copy Chapter

MNN uses a graph optimization + heterogeneous backend scheduling architecture.

Two inference APIs are available:
- Session API (low-level): Interpreter → createSession → runSession, operates on Tensor directly
- Module API (high-level, recommended): Module::load → onForward(VARP), Express-based dynamic graph. Used by LLM / Diffusion and most modern workloads

Key abstractions (see corresponding headers under source/core/):
- Interpreter / Session: model loading and inference session management
- Backend / Execution: hardware backend abstraction and per-op implementation (CPU/Metal/CUDA/OpenCL/Vulkan/...)
- Tensor: data container; internally uses NC4HW4 format (channels packed by 4 for SIMD)
- Op / Schema: FlatBuffers-defined operator descriptors (schema/default/*.fbs)

Op registration pattern: Schema definition → shape inference (source/shape/) → Geometry decomposition (optional) → Backend Execution implementation

LLM Subsystem Copy Chapter

MNN supports end-to-end LLM export and inference:

- Python export (transformers/llm/export/): HuggingFace model → MNN format. Core modules: llmexport.py entry point, utils/model_mapper.py (model field mapping), utils/model.py (unified LlmModel class), utils/transformers.py (Attention/Decoder/RoPE export)
- C++ inference (transformers/llm/engine/): llm.cpp (text inference), omni.cpp (multimodal: vision/audio), includes KVCache management and sampling strategies

Repository Structure Copy Chapter

| Directory | Description |
|-----------|-------------|
| include/MNN/ | Public C++ headers |
| source/core/ | Inference core (Interpreter, Session, Pipeline, Backend) |
| source/backend/ | Hardware backend implementations (cpu, arm82, metal, cuda, opencl, vulkan, ...) |
| source/shape/ | Shape inference |
| source/geometry/ | Geometry computation (op decomposition) |
| express/ | Express API (high-level dynamic graph, VARP) |
| schema/default/ | FlatBuffers schema (op definitions) |
| tools/converter/ | Model converter (ONNX/TF/Caffe → MNN) |
| transformers/llm/ | LLM export (Python) + inference engine (C++) |
| transformers/diffusion/ | Diffusion model support |
| pymnn/ | Python bindings |
| test/ | Test cases |
| skills/ | AI Agent Skills |

Coding Style Copy Chapter

- C++: Google Style variant, see .clang-format. 4-space indent, 120-char line width, attached braces. Class names PascalCase, functions camelCase, member variables mCamelCase. RTTI and exceptions disabled (-fno-rtti -fno-exceptions). Default standard: C++11.
- Python: Standard Python conventions
- Formatting: clang-format -i -style=file <file>

Build & Test Copy Chapter

Build C++ (with LLM)

mkdir build && cd build
cmake .. -DMNN_BUILD_LLM=ON -DMNN_LOW_MEMORY=ON && make -j$(nproc)
Common CMake options: MNN_BUILD_TEST, MNN_BUILD_CONVERTER, MNN_METAL, MNN_OPENCL,

MNN_VULKAN, MNN_CUDA, MNN_ARM82, MNN_BUILD_QUANTOOLS, MNN_SUPPORT_TRANSFORMER_FUSE

Full list: see option() declarations at the top of CMakeLists.txt
Unit tests

cd build && ./run_test.out
LLM export

cd transformers/llm/export
python llmexport.py --path /path/to/model --export mnn --hqq --dst_path ./MODEL
LLM test

cd build
./llm_demo /path/to/MODEL/config.json prompt.txt
LLM benchmark

./llm_bench -m /path/to/MODEL/config.json

Test suite includes: unit tests (run_test.out), model tests, conversion tests (ONNX/TF/TFLite/Torch), quantization tests, LLM tests, PyMNN tests. See test.sh and test/ directory for details.

Skills Copy Chapter

For the following tasks, read the Skill entry file first and execute step by step. Each step must pass its tests before proceeding.

After completing any skill-driven task, run the Retrospective skill to reflect on mistakes and update the skill with lessons learned.

| Skill | Entry File | Trigger |
|-------|-----------|---------|
| Support new LLM | skills/support-new-llm/SKILL.md | Add / adapt a new LLM model |
| Add new op | skills/add-new-op/SKILL.md | Add a new operator |
| ARM CPU optimization | skills/arm-cpu-optimize/SKILL.md | Optimize op performance on ARM CPU |
| Retrospective | skills/retrospective/SKILL.md | After any non-trivial task: reflect on mistakes, update relevant skills with lessons learned |

README.md Copy Chapter

!MNN
---


News 🔥 Copy Chapter

- [2026/02/13] MNN-Sana-Edit-V2 is now available at apps, offering cartoon-style photo editing based on Sana.
<p align="center">
<img width="80%" alt="Icon" src="https://meta.alicdn.com/data/mnn/assets/sana_show_case.jpg" style="margin: 0 10px;">
</p>

<details>
<summary> History News </summary>

- [2025/10/16] Support Qwen3-VL Series.
- [2025/06/11] New App MNN TaoAvatar released, you can talk with 3DAvatar offline with LLM, ASR, TTS, A2BS and NNR models all run local on your device!! MNN TaoAvatar
<p align="center">
<img width="20%" alt="Icon" src="https://meta.alicdn.com/data/mnn/avatar/avatar_demo.gif" style="margin: 0 10px;">
</p>

- [2025/05/12] android app support qwen2.5 omni 3b and 7b MNN Chat App.
<p align="center">
<img width="20%" alt="Icon" src="./apps/Android/MnnLlmChat/assets/image_home_new.jpg" style="margin: 0 10px;">
<img width="20%" alt="Icon" src="./apps/Android/MnnLlmChat/assets/image_sound_new.jpg" style="margin: 0 10px;">
<img width="20%" alt="Icon" src="./apps/Android/MnnLlmChat/assets/image_image_new.jpg" style="margin: 0 10px;">
</p>

- [2025/04/30] android app support qwen3 and dark mode MNN Chat App.
<p align="center">
<img width="20%" alt="Icon" src="https://meta.alicdn.com/data/mnn/qwen_3.gif" style="margin: 0 10px;">
</p>

- [2025/02/18] iOS multimodal LLM App is released MNN LLM iOS.
<p align="center">
<img width="20%" alt="Icon" src="./apps/iOS/MNNLLMChat/assets/introduction.gif" style="margin: 0 10px;">
</p>

- [2025/02/11] android app support for deepseek r1 1.5b.
<p align="center">
<img width="20%" alt="Icon" src="./apps/Android/MnnLlmChat/assets/deepseek_support.gif" style="margin: 0 10px;">
</p>

- [2025/01/23] We released our full multimodal LLM Android App:MNN-LLM-Android. including text-to-text, image-to-text, audio-to-text, and text-to-image generation.
<p align="center">
<img width="20%" alt="Icon" src="./apps/Android/MnnLlmChat/assets/image_home_new.jpg" style="margin: 0 10px;">
<img width="20%" alt="Icon" src="./apps/Android/MnnLlmChat/assets/image_diffusion_new.jpg" style="margin: 0 10px;">
<img width="20%" alt="Icon" src="./apps/Android/MnnLlmChat/assets/image_sound_new.jpg" style="margin: 0 10px;">
<img width="20%" alt="Icon" src="./apps/Android/MnnLlmChat/assets/image_image_new.jpg" style="margin: 0 10px;">
</p>
</details>

Intro Copy Chapter

MNN-LLM is a large language model runtime solution developed based on the MNN engine. The mission of this project is to deploy LLM models locally on everyone's platforms(Mobile Phone/PC/IOT). It supports popular large language models such as Qianwen, Baichuan, Zhipu, LLAMA, and others. MNN-LLM User guide

MNN-Diffusion is a stable diffusion model runtime solution developed based on the MNN engine. The mission of this project is to deploy stable diffusion models locally on everyone's platforms. MNN-Diffusion User guide

!architecture

Inside Alibaba, MNN works as the basic module of the compute container in the Walle System, the first end-to-end, general-purpose, and large-scale production system for device-cloud collaborative machine learning, which has been published in the top system conference OSDI’22. The key design principles of MNN and the extensive benchmark testing results (vs. TensorFlow, TensorFlow Lite, PyTorch, PyTorch Mobile, TVM) can be found in the OSDI paper. The scripts and instructions for benchmark testing are put in the path “/benchmark”. If MNN or the design of Walle helps your research or production use, please cite our OSDI paper as follows:

@inproceedings {proc:osdi22:walle,
author = {Chengfei Lv and Chaoyue Niu and Renjie Gu and Xiaotang Jiang and Zhaode Wang and Bin Liu and Ziqi Wu and Qiulin Yao and Congyu Huang and Panos Huang and Tao Huang and Hui Shu and Jinde Song and Bin Zou and Peng Lan and Guohuan Xu and Fei Wu and Shaojie Tang and Fan Wu and Guihai Chen},
title = {Walle: An {End-to-End}, {General-Purpose}, and {Large-Scale} Production System for {Device-Cloud} Collaborative Machine Learning},
booktitle = {16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)},
year = {2022},
isbn = {978-1-939133-28-1},
address = {Carlsbad, CA},
pages = {249--265},
url = {https://www.usenix.org/conference/osdi22/presentation/lv},
publisher = {USENIX Association},
month = jul,
}

Documentation and Workbench Copy Chapter

You can also read docs/README to build docs's html.

MNN Workbench could be downloaded from MNN's homepage, which provides pretrained models, visualized training tools, and one-click deployment of models to devices.

Key Features Copy Chapter

Lightweight Copy Section

Versatility Copy Section

High performance Copy Section

Ease of use Copy Section

The Architecture / Precision MNN supported is shown below:

- S ：Support and work well, deeply optimized, recommend to use
- A ：Support and work well, can use
- B ：Support but has bug or not optimized, no recommend to use
- C ：Not Support

| Architecture / Precision | | Normal | FP16 | BF16 | Int8 |
| --- | --- | --- | --- | --- | --- |
| CPU | Native | B | C | B | B |
| | x86/x64-SSE4.1 | A | C | C | A |
| | x86/x64-AVX2 | S | C | C | A |
| | x86/x64-AVX512 | S | C | C | S |
| | ARMv7a | S | S (ARMv8.2) | S | S |
| | ARMv8 | S | S (ARMv8.2) | S(ARMv8.6) | S |
| GPU | OpenCL | A | S | C | S |
| | Vulkan | A | A | C | A |
| | Metal | A | S | C | S |
| | CUDA | A | S | C | A |
| NPU | CoreML | A | C | C | C |
| | HIAI | A | C | C | C |
| | NNAPI | B | B | C | B |
| | QNN | C | B | C | C |

Tools Copy Chapter

Base on MNN (Tensor compute engine), we provided a series of tools for inference, train and general computation.

- MNN-Converter: Convert other models to MNN models for inference, such as Tensorflow(lite), Caffe, ONNX, Torchscripts. And do graph optimization to reduce computation.
- MNN-Compress: Compress model to reduce size and increase performance / speed
- MNN-Express: Support model with controlflow, use MNN's OP to do general-purpose computing.
- MNN-CV: An OpenCV-like library, but based on MNN and then much more lightweight.
- MNN-Train: Support train MNN model.

How to Discuss and Get Help From the MNN Community Copy Chapter

The group discussions are predominantly Chinese. But we welcome and will help English speakers.

Dingtalk discussion groups:

Group #4 (Available): 160170007549

Group #3 (Full)

Group #2 (Full): 23350225

Group #1 (Full): 23329087

Historical Paper Copy Chapter

The preliminary version of MNN, as mobile inference engine and with the focus on manual optimization, has also been published in MLSys 2020. Please cite the paper, if MNN previously helped your research:

@inproceedings{alibaba2020mnn,
author = {Jiang, Xiaotang and Wang, Huan and Chen, Yiliu and Wu, Ziqi and Wang, Lichuan and Zou, Bin and Yang, Yafeng and Cui, Zongyang and Cai, Yu and Yu, Tianhang and Lv, Chengfei and Wu, Zhihua},
title = {MNN: A Universal and Efficient Inference Engine},
booktitle = {MLSys},
year = {2020}
}

License Copy Chapter

Acknowledgement Copy Chapter

MNN refers to the following projects:
- Caffe
- flatbuffer
- gemmlowp
- Google Vulkan demo
- Halide
- Mace
- ONNX
- protobuffer
- skia
- Tensorflow
- ncnn
- paddle-mobile
- stb
- rapidjson
- pybind11
- pytorch
- bolt
- libyuv
- libjpeg
- opencv
- onnxruntime