MiniCPM Documentation

Technical Documentation: OpenBMB/MiniCPM

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (README.md) Copy Section

text Copy

[中文](https://github.com/OpenBMB/MiniCPM/blob/main/README-cn.md) | **English**

✨ Highlights Copy Chapter

We are releasing MiniCPM5-2B, the second model in the MiniCPM5 series, following MiniCPM5-1B. It is a dense 2B Transformer that scales up the same training recipe, built for on-device, local deployment, and resource-constrained scenarios, reaching 2B-class open-source SOTA.

🏆 2B-class open-source SOTA: compared with strong open-source models of similar size, MiniCPM5-2B achieves SOTA performance within this comparison set. It remains competitive with 4B-class models overall, while showing its advantages over models of comparable size in coding, mathematics, long-context understanding, tool use, and agentic tasks.

📂 Open High-Quality Data: Alongside the model, we are releasing the high-quality training datasets behind it as part of the UltraData family: UltraX, a high-quality web pre-training dataset; UltraData-Code, featuring L0–L3 tiered code data management to drive a significant leap in coding capabilities; UltraData-SFT-Agent-2609, comprising 500K agent training samples to enhance comprehensive on-device agent capabilities; and UltraData-RL-2609, with 80K+ high-quality RL training samples covering mathematics, code, general knowledge, and long-context reasoning.

🔥 Changelog Copy Chapter

• 📌 [2026.09.07] MiniCPM5-2B is released: a compact 2B-class dense model for on-device and resource-constrained use, paired with deployment / fine-tuning Agent Skills.
• [2026.05.19] MiniCPM5-1B is released: a compact 1B-class dense model for on-device and resource-constrained use, paired with deployment / fine-tuning Agent Skills.
• [2026.02.11] MiniCPM-SALA is released: a sparse-and-linear hybrid attention model for million-token context modeling and efficient inference.
• [2025.09.05] MiniCPM4.1 series is released: a trainable sparse-attention model with hybrid reasoning.
• [2025.06.06] MiniCPM4 is released: an end-side model with over 5x generation acceleration on typical edge chips.

Older entries (2024 + InfLLM-V2 paper)

• [2025.09.29] InfLLM-V2 paper is released! We can train a sparse attention model with only 5B long-text tokens.
• [2024.09.05] We release MiniCPM3-4B! This model outperforms Phi-3.5-mini-instruct and GPT-3.5-Turbo-0125 and is comparable to several models with 7B-9B parameters like Llama3.1-8B-Instruct, Qwen2-7B-Instruct, and GLM-4-9B-Chat.
• [2024.07.05] Released MiniCPM-S-1B! This model achieves an average sparsity of 87.89% in the FFN layer, reducing FFN FLOPs by 84%, while maintaining downstream task performance.
• [2024.04.11] Released MiniCPM-2B-128k, MiniCPM-MoE-8x2B and MiniCPM-1B! Click here to read our technical blog.
• [2024.02.01] Released MiniCPM-2B! This model performs similarly to Mistral-7B on public benchmarks (with better performance in Chinese, math, and code abilities) and overall outperforms models like Llama2-13B, MPT-30B, and Falcon-40B.

🧭 Quick Links Copy Chapter

• ✨ Highlights
• 🔥 Changelog
• 📦 Model Downloads
• 🚀 MiniCPM5-2B
  • Introduction
  • Evaluation Results
  • Training Recipe
    • What does RL + OPD bring?
  • Quickstart
  • Deployment and Fine-tuning Cookbooks and Agent Skills
  • Other Supported Frameworks
• 🚀 MiniCPM5-1B
• 🧪 MiniCPM-SALA
• ⚡ MiniCPM4 & MiniCPM4.1 Series
• Legacy topics →: BitCPM4 quantization, MiniCPM4 applications
• 📄 LICENSE · 🏛 Institutions · 📚 Citation

📦 Model Downloads Copy Chapter

Current release: MiniCPM5-2B / MiniCPM5-1B（BF16 / GGUF / MLX）：

MiniCPM5-2B

MiniCPM5-1B

Other key releases:

📋 Click to view earlier MiniCPM releases: 4, BitCPM, applications, MiniCPM3 / 2B / 1B

Earlier flagships:

MiniCPM4.1 quantized & speculative variants:

BitCPM4 ternary-quantized + MiniCPM4 Applications:

MiniCPM4 Eagle speculative decoding, QAT, and pre-2025 releases:

🚀 MiniCPM5-2B Copy Chapter

Introduction Copy Section

MiniCPM5-2B is the second checkpoint in the MiniCPM5 series, scaling the MiniCPM5-1B recipe up to 2B parameters for users who can afford a larger footprint in exchange for stronger capability. It is designed for local assistants, coding agents, tool-use workflows, and reasoning scenarios where a compact model is preferred. The model keeps a small deployment footprint while providing native long-context support.

Evaluation Results Copy Section

We compare MiniCPM5-2B with strong open-source models in the same size class, including LFM2.5-2.6B, Qwen3.5-2B, and Gemma-4-E2B-it, while also listing larger models such as Qwen3.5-4B, granite-4.2-3B, Nemotron-3-Nano-4B, Gemma-4-E4B-it, and LFM2.5-8B-A1B for reference.

Within this comparison set, MiniCPM5-2B reaches 2B-class open-source SOTA with an average score of 53.9, and also exceeds all of the larger models included here (the highest is 51.1). Its advantages are most visible in code reasoning, math reasoning, long-context understanding, tool use, and multiple agentic tasks.

Training Recipe Copy Section

The training of MiniCPM5-2B is a full-stack practice of UltraData Tiered Data Management, covering three stages: base training, mid-training, and post-training.

During base training, the model goes through stable training and decay training to build core language capability and training stability. It then enters mid-training to further strengthen target capabilities and adapt to the target data distribution. The training corpus is released alongside the model as Ultra-FineWeb, Ultra-FineWeb-L3, UltraX, UltraData-Code and UltraData-Math.

During post-training, we proceed in three steps: SFT, RL, and OPD. We first use 400B tokens of deep-thinking SFT to establish deep-thinking and general chat abilities; the SFT data is released as UltraData-SFT-2605 and Agent SFT data as UltraData-SFT-Agent-2609. We then train specialized RL teachers for math, code, agentic tasks, writing, and related domains(with the corresponding data also open-sourced as UltraData-RL-2609), and use On-Policy Distillation (OPD) to distill these teachers back into one release model.

What does RL + OPD bring?

RL + OPD is a key part of MiniCPM5-2B post-training. During the RL stage, we adopted the critic-based algorithm described in JustRL II, substantially improving training stability and achieving significant gains across multiple domains. On the benchmarks listed below, RL + OPD improves reasoning and general capabilities by an average of ↑10.96 points, and agentic capabilities by ↑6.96 points.

OPD merges the capabilities of 16 expert models produced by RL training, including 5 agentic expert models. At each response position, we compute the full-vocabulary reverse KL divergence between student and teacher logits as the advantage estimate, replacing the original verification-based advantage. OPD directly reuses the prompts used to train each RL teacher as distillation data, so no additional corpus construction is required.

Quickstart Copy Section

We recommend using the following sets of sampling parameters for generation: temperature=1.0, top_p=0.95, min_p=0.0.

If you encounter repetitive outputs, try: temperature=1.0, top_p=0.95, min_p=0.0, repetition_penalty=1.05.

Please note that the support for sampling parameters varies according to inference frameworks.

vLLM

pip install "vllm>=0.21"
vllm serve openbmb/MiniCPM5-2B --port 8000

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openbmb/MiniCPM5-2B",
    "messages": [{"role": "user", "content": "Who are you? Please briefly introduce yourself."}],
    "max_tokens": 128,
    "temperature": 1.0, "top_p": 0.95
  }'

SGLang

pip install "sglang[srt]>=0.5.16"
python -m sglang.launch_server --model-path openbmb/MiniCPM5-2B --port 30000

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openbmb/MiniCPM5-2B",
    "messages": [{"role": "user", "content": "Who are you? Please briefly introduce yourself."}],
    "max_tokens": 128,
    "temperature": 1.0, "top_p": 0.95
  }'

Speculative decoding (DSpark): we also release MiniCPM5-2B-DSpark, a DSpark draft model trained for MiniCPM5-2B. Enable it in SGLang to accelerate decoding while keeping the target model's outputs unchanged:

python -m sglang.launch_server \
  --model-path openbmb/MiniCPM5-2B \
  --trust-remote-code \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path openbmb/MiniCPM5-2B-DSpark \
  --speculative-dspark-block-size 7 \
  --port 30000

Llama.cpp

llama-server -m MiniCPM5-2B-F16.gguf -a MiniCPM5-2B --port 8080 -ngl 99 -c 8192 --jinja

-c 8192 sets the context length. You can adjust this value as needed.

curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "MiniCPM5-2B",
        "messages": [{"role": "user", "content": "1+1=?"}],
        "temperature": 1.0, "top_p": 0.95, "min_p": 0.0, "max_tokens": 256
    }'

In llama.cpp, the default min_p=0.05 can lead to repetitive output: it filters out tokens whose probability is below 5% of the highest-probability token, potentially discarding the exact tokens needed to break out of a repetition loop. To prevent this, we set min_p=0.0.

Transformers

pip install -U "transformers>=5.6" accelerate torch

from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "openbmb/MiniCPM5-2B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)
messages = [{"role": "user", "content": "Who are you? Please briefly introduce yourself."}]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Tool Calling Copy Section

For tool / function calling, SGLang is the recommended backend. MiniCPM5-2B emits XML-style tool calls and SGLang's built-in minicpm5 parser converts them to OpenAI-compatible tool_calls natively:

python -m sglang.launch_server --model-path openbmb/MiniCPM5-2B --port 30000 \
    --tool-call-parser minicpm5      # or: --tool-call-parser auto

GitHub Cookbooks and Agent Skills Copy Section

MiniCPM5-2B uses the standard LlamaForCausalLM architecture, so mainstream inference engines can load it directly: no custom kernels, no model-code fork. For step-by-step deployment and fine-tuning instructions, use the GitHub cookbooks below. Agent Skills are linked as GitHub resources for users working with Cursor / Claude Code style coding agents.

Deployment

Fine-tuning

Other Supported Frameworks

In addition to the deployment and fine-tuning frameworks listed above, MiniCPM5-2B is also supported by FlagOS for multi-chip deployment.

FlagOS Overview

To enable large-scale deployment across different AI chips, Beijing Zhiyuan Research Institute, together with numerous research institutions, chip manufacturers, system vendors, and algorithm and software organizations both domestically and internationally, jointly initiated and established the FlagOS Open Source Community.

The FlagOS community is dedicated to building a unified, open-source system software stack for various AI chips, encompassing core open-source projects such as a large-scale operator library, a unified AI compiler, parallel training and inference frameworks, and a unified communication library. It aims to create an open technology ecosystem connecting the “model-system-chip” layers. By enabling “develop once, deploy across chips”, FlagOS unlocks the computational potential of hardware, breaks down the ecosystem silos between different chip software stacks, and effectively reduces migration costs for developers.The FlagOS community fosters an AI hardware and software ecosystem, overcomes single-vendor closed-source monopolies, promotes widespread deployment of AI hardware technologies, and is committed to rooted in China while embracing global collaboration.

Official website express: https://flagos.io

FlagOS multi-chip support and usage

FlagOS: Supporting Multiple AI Chips

Thanks to FlagOS’s unified multi-chip AI system software stack, MiniCPM5-2B was adapted to 9 different AI chips in an extremely short time. Currently, the multi-chip version of MiniCPM5-2B has been released on FlagRelease, FlagOS’s platform for automatic migration, adaptation, and deployment of large models across multi-architecture AI chips. Details are as follows:

FlagOS Usage

FlagOS Performance Acceleration on Nvidia

From FlagRelease (Recommendation)

FlagRelease is a platform developed by the FlagOS team for automatic migration, adaptation, and deployment of large models across multi-architecture AI chips. The multi-chip version of MiniCPM5-2B has already been released on FlagRelease. All necessary software packages are pre-installed on the platform, so users do not need to install anything.

FlagRelease Quick Start

From Scratch

• Dependencies: Python 3.12, GLIBC 2.39, GLIBCXX 3.4.33, CXXABI 1.3.15

Installing the FlagOS Operator Library

Official Repository: https://github.com/flagos-ai/FlagGems

pip install flag-gems==4.2.1rc0
pip install triton==3.5.1

Activating Acceleration

You can enable flagGems acceleration by adding the import of flagGems in the source code of vllm where inference is performed.

import flag_gems
flag_gems.enable(record=True, once=True, path="/root/gems.txt")

vllm serve ${model_path} \
--trust-remote-code \
--dtype bfloat16 \
--enforce-eager \
--port ${Port} \
--served-model-name ${model_name} \
--gpu-memory-utilization 0.85

Using FlagOS Unified Multi-Chip Backend Plugin

vllm-plugin-FL is a plugin built for the vLLM inference/service framework. Developed on top of FlagOS’s unified multi-chip backend, it is designed to extend vLLM’s capabilities and performance across a variety of hardware environments.

Using vllm-plugin-FL

🚀 MiniCPM5-1B Copy Chapter

MiniCPM5-1B is the first model in the MiniCPM5 series. It is a dense 1B Transformer built for on-device, local deployment, and resource-constrained scenarios, reaching 1B-class open-source SOTA.

Click to expand: highlights, introduction, evaluation, inference setup

Highlights Copy Section

🏆 1B-class open-source SOTA: compared with strong open-source models in the same size class, MiniCPM5-1B reaches SOTA within this comparison set. Its advantage is most visible in agentic tool use, code generation, and difficult reasoning.

🧠 Hybrid Reasoning: built-in `` chat template, switch via enable_thinking. The same checkpoint serves as both a fast assistant and a deliberate reasoner.

🛠️ Deployment / Fine-tuning Resources: the MiniCPM GitHub repo provides single-page cookbooks and Agent Skills for major inference backends and fine-tuning frameworks.

🐱 Desktop Pet: a local-LLM desktop pet driven by MiniCPM5-1B.

Introduction Copy Section

MiniCPM5-1B is the first checkpoint in the MiniCPM5 series. It is designed for local assistants, coding agents, tool-use workflows, and reasoning scenarios where a compact model is preferred. The model keeps a small deployment footprint while providing native long-context support and both Think / No Think chat modes through the same checkpoint.

Evaluation Results Copy Section

We compare MiniCPM5-1B with strong open-source models in the same size class, including LFM2.5-1.2B-Thinking, Qwen3-0.6B/think and Qwen3.5-0.8B/think. These are capable baselines; within this comparison set, MiniCPM5-1B reaches 1B-class open-source SOTA, with its advantage most visible in tool use, code generation, and difficult reasoning. This makes it a practical choice for local coding agents, tool assistants, and reasoning assistants.

Training Recipe Copy Section

The training of MiniCPM5-1B is a full-stack practice of UltraData Tiered Data Management, covering three stages: base training, mid-training, and post-training.

During base training, the model goes through stable training and decay training to build core language capability and training stability. It then enters mid-training to further strengthen target capabilities and adapt to the target data distribution. The training corpus is released alongside the model as Ultra-FineWeb, Ultra-FineWeb-L3, and UltraData-Math.

During post-training, we proceed in three steps: SFT, RL, and OPD. We first use 200B tokens of deep-thinking SFT and 200B tokens of hybrid-thinking SFT to establish deep-thinking, hybrid-thinking, and general chat abilities; the SFT data is released as UltraData-SFT-2605. We then train specialized RL teachers for math, code, closed-book QA, writing, and related domains, and use On-Policy Distillation (OPD) to distill these teachers back into one release model.

What does RL + OPD bring?

RL + OPD is a key part of MiniCPM5-1B post-training. On math, code and instruction-following tasks, RL + OPD raises the average score by ↑16 points while cutting the share of responses that hit the max-tokens budget by ↓29 percentage points. The figures below show the two-stage Reasoning RL pipeline, score gains, and the drop in overlong responses.

RL combines complementary training signals for reasoning, closed-book QA, writing, instruction following, long-context understanding, and general dialogue. Reasoning RL is based on DAPO-Math-17k (inspired by JustRL's minimalist recipe) and uses a two-stage length schedule to reduce overlong responses while improving reasoning accuracy. We also use TriviaQA, NQ-Open, LongWriter-Zero-RLData, synthesized verifiable RLVR data, and pair-wise RLHF signals to improve reliability, instruction following, and user experience.

OPD builds on Thinking Machines Lab's On-Policy Distillation and incorporates implementation improvements from Rethinking On-Policy Distillation. In the RL framework, we use reverse KL divergence as the advantage estimate, replacing the original verification-based advantage. At each response position, we take top-k logits from both the student and teacher models, compute reverse KL on the union of the two token sets, and balance the accuracy of the RKL signal with training efficiency. OPD reuses the in-domain prompts used to train each RL teacher as distillation data, so no additional data curation is required.

Quickstart Copy Section

For the three most common inference paths, you can start a service or run local inference as follows:

vLLM (OpenAI-compatible server, NVIDIA GPU):

pip install "vllm>=0.21" && vllm serve openbmb/MiniCPM5-1B --port 8000

Test request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openbmb/MiniCPM5-1B",
    "messages": [{"role": "user", "content": "Who are you? Please briefly introduce yourself."}],
    "max_tokens": 128,
    "temperature": 0.7
  }'

SGLang (OpenAI-compatible server, NVIDIA GPU):

pip install "sglang[srt]>=0.5.12" && python -m sglang.launch_server --model-path openbmb/MiniCPM5-1B --port 30000

Test request:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openbmb/MiniCPM5-1B",
    "messages": [{"role": "user", "content": "Who are you? Please briefly introduce yourself."}],
    "max_tokens": 128,
    "temperature": 0.7
  }'

Transformers (local Python inference, CPU or GPU):

pip install -U "transformers>=5.6" accelerate torch

Local inference test:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "openbmb/MiniCPM5-1B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

messages = [{"role": "user", "content": "Who are you? Please briefly introduce yourself."}]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Recommended chat template sampling:

For other backends (llama.cpp / Ollama / LM Studio / MLX), see the cookbooks table below.

With tool calling

For tool / function calling, SGLang is the recommended backend — MiniCPM5-1B emits XML-style tool calls and SGLang's built-in minicpm5 parser converts them to OpenAI-compatible tool_calls natively:

python -m sglang.launch_server --model-path openbmb/MiniCPM5-1B --port 30000 \
    --tool-call-parser minicpm5      # or: --tool-call-parser auto

Deployment and Fine-tuning Cookbooks and Agent Skills Copy Section

MiniCPM5-1B uses the standard LlamaForCausalLM architecture, so mainstream inference engines can load it directly: no custom kernels, no model-code fork. To help developers reproduce deployment and fine-tuning workflows, we provide single-page cookbooks paired with Cursor Agent Skills: the cookbooks are for manual execution, while Agent Skills let Cursor / Claude Code agents choose the route based on the target backend, hardware, and data path.

The two top-level skills cover deployment and fine-tuning:

In Cursor / Claude Code, you can call them like this: the agent reads the top-level skill, selects the matching sub-skill and cookbook based on the target backend, hardware, and data path, then runs the command and reports back.

@minicpm5-deploy   serve openbmb/MiniCPM5-1B with vLLM on port 8000
@minicpm5-finetune use unsloth + LoRA on /data/my_chat.jsonl, write to ./out

The tables below list the cookbook and sub-skill for each inference backend and fine-tuning framework. Quantized models are not listed as standalone backends; they are described under the inference backend that can load each format.

Inference Deployment (8 backends)

Fine tuning (5 frameworks)

Other Supported Frameworks Copy Section

In addition to the deployment and fine-tuning frameworks listed above, MiniCPM5-1B is also supported by FlagOS for multi-chip deployment.

Desktop Pet Copy Section

We also ship OpenBMB/MiniCPM-Desk-Pet, a desktop pet driven locally by MiniCPM5-1B. It uses a thin llama.cpp llama-server sidecar to load the GGUF model and serves an OpenAI-compatible local endpoint to an Electron pet UI.

The pet supports Apple Silicon / NVIDIA GPU / CPU paths, can work with coding agents such as Cursor, Claude Code, and Codex, and supports LoRA persona switching.

• User install: grab Clawd-on-Desk-*-arm64.dmg from Releases, then follow the onboarding flow for environment checks, model download, and sidecar startup.
• Developer run: git clone [email protected]:OpenBMB/MiniCPM-Desk-Pet.git && ./go.sh — see MiniCPM-Desk-Pet/README.md for the full setup.

The pet UI layer is forked from @rullerzhou-afk/clawd-on-desk (AGPL-3.0). On top of the upstream pet runtime, animation packs, and multi-agent integrations, we add the local MiniCPM5-1B sidecar, onboarding flow, and LoRA persona switching. Full attribution in NOTICE.md.

🧪 MiniCPM-SALA Copy Chapter

First large-scale Sparse + Linear Attention hybrid for million-token context (2026-02). HuggingFace · ModelScope

Click to expand: highlights, evaluation, inference setup

Highlights

MiniCPM-SALA (Sparse Attention and Linear Attention) is the first large-scale hybrid model effectively integrating sparse and linear attention for million-token context modeling

✅ Hybrid Architecture: combines 25% Sparse Attention (InfLLM-v2) for long-context modeling with 75% Linear Attention (Lightning Attention) for global efficiency.

✅ Inference Efficiency: achieves 3.5× inference speed and lower KV-cache overhead compared with dense baselines.

✅ Million-Token Context: uses HyPE (Hybrid Positional Embedding) to scale to 1M+ tokens while maintaining length generalization.

✅ HALO Adaptation: uses Hybrid Attention via Layer Optimization (HALO), a distillation recipe that transfers dense attention capabilities to the hybrid architecture and mitigates the degradation often seen in pure linear models.

Introduction

MiniCPM-SALA is an efficient hybrid model in which 25% of the layers adopt InfLLM-V2 and the remaining 75% utilize Lightning Attention. This architecture enables inference of one million tokens on consumer GPUs such as the NVIDIA RTX 5090.

• SALA Hybrid Attention Mechanism

  • Integrates 25% InfLLM-V2 and 75% Lightning Attention, effectively leveraging the granular focus of sparse attention for local details and the high efficiency of linear attention for broad context.

• Transformer-to-Hybrid Continue Training

  • Circumvents the inefficiencies of cold-start training by performing an architectural transformation on the pre-trained weights, thereby reducing the total training budget to approximately 25% relative to training a comparable model from scratch.

• HyPE (Hybrid Positional Encoding)

  • Balances short-context and long-context performance, maintaining general capabilities (e.g., knowledge, mathematics, and coding) close to full-attention models such as Qwen3-8B while scoring higher on multiple long-context benchmarks.

• Efficient Inference on Long Sequences

  • Achieves up to 3.5x the inference speed of Qwen3-8B at a sequence length of 256K tokens on A6000D, supports inference at context lengths of up to 1M tokens on both NVIDIA A6000D and 5090 GPUs, whereas Qwen3-8B fails at this length due to out-of-memory (OOM) errors.

Evaluation Results Copy Section

Efficiency Evaluation

We benchmarked MiniCPM-SALA (9B) against Qwen3-8B on NVIDIA A6000D and RTX 5090 GPUs to evaluate inference speed and memory efficiency. MiniCPM-SALA achieves up to a 2.5x speedup in time-to-first-token (TTFT) and reduces the memory pressure of full-attention architectures at ultra-long lengths. In this setup, Qwen3-8B runs into OOM errors at extended lengths, while MiniCPM-SALA can process 1M-token contexts on a single consumer-grade RTX 5090.

Long-Context Evaluation

MiniCPM-SALA scores higher than the tested open-source LLMs of similar scale on most long-context benchmarks. It achieves the highest scores in the RULER and NoLiMa tests at all context lengths up to 128K, with an overall average score of 38.97.

Ultra-long Context Evaluation

MiniCPM-SALA shows effective length extrapolation, maintaining a score of 81.6 at a 2048K context length despite being trained on up to 520K tokens. The model does this without auxiliary techniques like YaRN, likely due to its NoPE configuration in sparse attention layers.

Standard Evaluation

MiniCPM-SALA achieves an average score of 76.53 across standard benchmarks, outperforming comparable models such as Qwen3-8B and Falcon-H1R-7B. The architecture maintains robust performance in Knowledge, Code, and Math.

Inference Copy Section

Recommended inference setting: Temperature=0.9.

HuggingFace

Our model is readily compatible with 🤗 Hugging Face transformers. You can perform inference with our model as follows:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "openbmb/MiniCPM-SALA"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True, device_map="auto")
model.eval()

prompts = ["My name is", "The capital of China is"]
with torch.no_grad():
    inputs = tokenizer(prompts, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs)
output_texts = tokenizer.batch_decode(outputs)
print(output_texts)

SGLang

Requirements

• CUDA 12.x or higher
• gcc / g++ compiler
• uv package manager (script will check)

Installation

# Clone repository
git clone -b minicpm_sala https://github.com/OpenBMB/sglang.git
cd sglang

# One-click installation (creates venv and compiles all dependencies)
bash install_minicpm_sala.sh

# Or specify PyPI mirror
bash install_minicpm_sala.sh https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple

The installation script performs the following steps:

1. Creates sglang_minicpm_sala_env virtual environment (Python 3.12)
2. Clones dependencies to 3rdparty/ (infllmv2) and initializes submodules (sparse_kernel)
3. Installs MiniCPM-SALA (current repo)
4. Compiles and installs infllmv2_cuda_impl
5. Compiles and installs sparse_kernel
6. Installs tilelang & flash-linear-attention

Usage

# Activate environment
source sglang_minicpm_sala_env/bin/activate

# Launch Inference Server (Replace MODEL_PATH with actual path)
MODEL_PATH=/path/to/your/MiniCPM-SALA

python3 -m sglang.launch_server \
    --model ${MODEL_PATH} \
    --trust-remote-code \
    --disable-radix-cache \
    --attention-backend minicpm_flashinfer \
    --chunked-prefill-size 8192 \
    --max-running-requests 32 \
    --skip-server-warmup \
    --port 31111 \
    --dense-as-sparse

Manual Installation

If the script doesn't work for you, follow these steps:

# 0. Ensure uv is installed
pip install uv

# 1. Create venv
uv venv --python 3.12 sglang_minicpm_sala_env
source sglang_minicpm_sala_env/bin/activate

# 2. Install SGLang
uv pip install --upgrade pip setuptools wheel
uv pip install -e ./python[all]

# 3. Compile CUDA Extensions
# (Ensure dependencies are cloned to 3rdparty/)
cd 3rdparty/infllmv2_cuda_impl && python setup.py install && cd ../..
cd 3rdparty/sparse_kernel && python setup.py install && cd ../..

# 4. Install extra deps
uv pip install tilelang flash-linear-attention

Q&A

Q: CUDA extension compilation failed?

• Ensure CUDA 12+ is installed (nvcc --version).
• Ensure gcc / g++ are available.
• If CXX is set to clang++ -pthread, manually export CXX=g++.

⚡ MiniCPM4 and MiniCPM4.1 Series Copy Chapter

8B-scale trainable sparse attention with hybrid reasoning (2025-09 / 2025-06). MiniCPM4.1-8B · MiniCPM4-8B · ModelScope

Click to expand: highlights, evaluation, inference (HF / vLLM / SGLang / CPM.cu / llama.cpp / Ollama)

Highlights

MiniCPM 4.1-8B is the first open-source reasoning LLM with trainable sparse attention:

✅ Strong Reasoning Capability: Surpasses similar-sized models on 15 tasks!

✅ Fast Generation: 3x decoding speedup for reasoning

✅ Efficient Architecture: Trainable sparse attention, frequency-ranked speculative decoding

Introduction

MiniCPM4 and MiniCPM4.1 series are large language models designed for end-side devices, with efficiency optimizations across model architecture, training data, training algorithms, and inference systems.

• 🏗️ Efficient Model Architecture:

  • InfLLM-V2 -- Trainable Sparse Attention Mechanism: Adopts a trainable sparse attention mechanism architecture where each token only needs to compute relevance with less than 5% of tokens in 128K long text processing, significantly reducing computational overhead for long texts (InfLLM-V2 Training Kernels)

• 🧠 Efficient Learning Algorithms:

  • Model Wind Tunnel 2.0 -- Efficient Predictable Scaling: Introduces scaling prediction methods for performance of downstream tasks, enabling more precise model training configuration search
  • BitCPM -- Ternary Quantization: Compresses model parameter bit-width to 3 values, achieving 90% model bit-width reduction
  • Efficient Training Engineering Optimization: Adopts FP8 low-precision computing technology combined with Multi-token Prediction training strategy

• 📚 High-Quality Training Data:

  • UltraClean -- High-quality Pre-training Data Filtering and Generation: Builds iterative data cleaning strategies based on efficient data verification, open-sourcing high-quality Chinese and English pre-training dataset UltraFinweb
  • UltraChat v2 -- High-quality Supervised Fine-tuning Data Generation: Constructs large-scale high-quality supervised fine-tuning datasets covering multiple dimensions including knowledge-intensive data, reasoning-intensive data, instruction-following data, long text understanding data, and tool calling data

• ⚡ Efficient Inference and Deployment System:

  • CPM.cu -- Lightweight and Efficient CUDA Inference Framework: Integrates sparse attention, model quantization, and speculative sampling to achieve efficient prefilling and decoding (Inference Kernels and Framework)
  • ArkInfer -- Cross-platform Deployment System: Supports efficient deployment across multiple backend environments, providing flexible cross-platform adaptation capabilities

Evaluation Results Copy Section

Efficiency Evaluation

On two typical end-side chips, Jetson AGX Orin and RTX 4090, MiniCPM4 and MiniCPM4.1 show faster processing speed than similar-size models in long-text processing tasks. As text length increases, the speed gains become more pronounced. On Jetson AGX Orin, compared with Qwen3-8B, MiniCPM4 and MiniCPM4.1 achieve approximately 7x decoding speed improvement.

MiniCPM4.1 achieves 3x decoding speed improvement in reasoning.

Comprehensive Evaluation

MiniCPM4 launches end-side versions with 8B and 0.5B parameter scales, both showing competitive performance in their respective categories.

MiniCPM4.1 launches an 8B end-side version with competitive performance in deep reasoning mode.

Long Text Evaluation

MiniCPM4 is pre-trained on 32K long texts and achieves length extension through YaRN. In the 128K needle-in-a-haystack task, MiniCPM4 maintains stable performance. MiniCPM4.1 is pre-trained on 64K long texts and also uses YaRN for length extension, with stable performance on the 128K needle-in-a-haystack task.

Inference Copy Section

MiniCPM 4.1 can be used with the following frameworks: Huggingface Transformers, SGLang, vLLM, and CPM.cu. For inference efficiency, CPM.cu is a good first option.

MiniCPM4/MiniCPM4.1 supports both dense attention inference and sparse attention inference modes, where vLLM and SGLang currently only support dense inference mode. If you want to use sparse inference mode, please use Huggingface Transformers and CPM.cu.

• Dense attention inference: vLLM, SGLang, Huggingface Transformers
• Sparse attention inference: Huggingface Transformers, CPM.cu

Hybrid Reasoning Mode

MiniCPM4.1 supports hybrid reasoning mode, which can be used in both deep reasoning mode and non-reasoning mode. To enable hybrid reasoning mode. User can set enable_thinking=True in tokenizer.apply_chat_template to enable hybrid reasoning mode, and set enable_thinking=False to enable non-reasoning mode. Similarly, user can directly add /no_think at the end of the query to enable non-reasoning mode. If not add any special token or add /think at the end of the query, the model will enable reasoning mode.

# Enable reasoning mode
prompt_text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True
)
# Enable non-reasoning mode
prompt_text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False
)

HuggingFace

• Inference with Dense Attention

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
torch.manual_seed(0)

path = 'openbmb/MiniCPM4.1-8B'
device = "cuda"
tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map=device, trust_remote_code=True)

# User can directly use the chat interface
# responds, history = model.chat(tokenizer, "Write an article about Artificial Intelligence.", temperature=0.7, top_p=0.7)
# print(responds)

# User can also use the generate interface
messages = [
    {"role": "user", "content": "Write an article about Artificial Intelligence."},
]
prompt_text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
model_inputs = tokenizer([prompt_text], return_tensors="pt").to(device)

model_outputs = model.generate(
    **model_inputs,
    max_new_tokens=32768,
    top_p=0.95,
    temperature=0.6
)
output_token_ids = [
    model_outputs[i][len(model_inputs[i]):] for i in range(len(model_inputs['input_ids']))
]

responses = tokenizer.batch_decode(output_token_ids, skip_special_tokens=True)[0]
print(responses)

• Inference with Sparse Attention
  This model supports InfLLM v2, a sparse attention mechanism designed for efficient long-sequence inference. It requires the infllmv2_cuda_impl library.

You can install it by running the following command:

git clone -b feature_infer https://github.com/OpenBMB/infllmv2_cuda_impl.git
cd infllmv2_cuda_impl
git submodule update --init --recursive
pip install -e . # or python setup.py install

To enable InfLLM v2, you need to add the sparse_config field in config.json:

{
    ...,
    "sparse_config": {
        "kernel_size": 32,
        "kernel_stride": 16,
        "init_blocks": 1,
        "block_size": 64,
        "window_size": 2048,
        "topk": 64,
        "use_nope": false,
        "dense_len": 8192
    }
}

These parameters control the behavior of InfLLM v2:

• kernel_size (default: 32): The size of semantic kernels.
• kernel_stride (default: 16): The stride between adjacent kernels.
• init_blocks (default: 1): The number of initial blocks that every query token attends to. This ensures attention to the beginning of the sequence.
• block_size (default: 64): The block size for key-value blocks.
• window_size (default: 2048): The size of the local sliding window.
• topk (default: 64): The specifies that each token computes attention with only the top-k most relevant key-value blocks.
• use_nope (default: false): Whether to use the NOPE technique in block selection for improved performance.
• dense_len (default: 8192): Since Sparse Attention offers limited benefits for short sequences, the model can use standard (dense) attention for shorter texts. The model will use dense attention for sequences with a token length below dense_len and switch to sparse attention for sequences exceeding this length. Set this to -1 to always use sparse attention regardless of sequence length.

• Long Context Extension
  MiniCPM4.1 natively supports context lengths of up to 65,536(64k) tokens. For conversations where the total length (including both input and output) significantly exceeds this limit, we recommend using RoPE scaling techniques for effective handling of long texts. By modifying the LongRoPE factor, the model can stably handle context lengths of up to 131,072 tokens.

You can apply the LongRoPE factor modification by modifying the model files. Specifically, in the config.json file, adjust the rope_scaling fields.

/* Detailed source-code truncated for AI context efficiency. */

vLLM

Speculative Decoding

For accelerated inference with speculative decoding using vLLM, follow these steps:

1. Download MiniCPM4.1 Draft Model

First, download the MiniCPM4.1 draft model:

cd /your_path
git clone https://huggingface.co/openbmb/MiniCPM4.1-8B-Eagle3

2. Install EAGLE3-Compatible vLLM

The EAGLE3 vLLM PR has been submitted. For now, use our repository for installation:

git clone https://github.com/LDLINGLINGLING/vllm.git
cd vllm 
pip install -e .

3. Launch vLLM Server with Speculative Decoding

Start the vLLM inference server with speculative decoding enabled. Make sure to update the model path in the speculative-config to point to your downloaded MiniCPM4_1-8B-Eagle3-bf16 folder:

VLLM_USE_V1=1 \
vllm serve openbmb/MiniCPM4.1-8B \
--seed 42 \
--trust-remote-code \
--speculative-config '{
  "model": "your/path/MiniCPM4_1-8B-Eagle3-bf16",
  "num_speculative_tokens": 3,
  "method": "eagle3",
  "draft_tensor_parallel_size": 1
}'

4. Client Usage Example

The client usage remains the same for both standard and speculative decoding:

import openai

client = openai.Client(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="openbmb/MiniCPM4.1-8B",
    messages=[
        {"role": "user", "content": "Write an article about Artificial Intelligence."},
    ],
    temperature=0.6,
    max_tokens=32768,
    extra_body=dict(add_special_tokens=True),  # Ensures special tokens are added for chat template
    
)

print(response.choices[0].message.content)

vLLM Configuration Parameters

• VLLM_USE_V1=1: Enables vLLM v1 API
• --speculative-config: JSON configuration for speculative decoding
  • model: Path to the draft model for speculation
  • num_speculative_tokens: Number of speculative tokens (default: 3)
  • method: Speculative decoding method (eagle3)
  • draft_tensor_parallel_size: Tensor parallel size for draft model (default: 1)
• --seed: Random seed for reproducibility
• --trust-remote-code: Allow execution of remote code for custom models

Standard Inference (Without Speculative Decoding)

For now, you need to install the latest version of vLLM.

pip install -U vllm \
    --pre \
    --extra-index-url https://wheels.vllm.ai/nightly

Then you can inference MiniCPM4.1-8B with vLLM:

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

model_name = "openbmb/MiniCPM4.1-8B"
prompt = [{"role": "user", "content": "Write an article about Artificial Intelligence."}]

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
input_text = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True)

llm = LLM(
    model=model_name,
    trust_remote_code=True,
    max_num_batched_tokens=65536,
    dtype="bfloat16", 
    gpu_memory_utilization=0.8, 
)
sampling_params = SamplingParams(top_p=0.95, temperature=0.6, max_tokens=32768)

outputs = llm.generate(prompts=input_text, sampling_params=sampling_params)

print(outputs[0].outputs[0].text)

Also, you can start the inference server by running the following command:

Note: In vLLM's chat API, add_special_tokens is False by default. This means important special tokens—such as the beginning-of-sequence (BOS) token—will not be added automatically. To ensure the input prompt is correctly formatted for the model, you should explicitly set extra_body={"add_special_tokens": True}.

vllm serve openbmb/MiniCPM4.1-8B --trust-remote-code

Then you can use the chat interface by running the following code:

import openai

client = openai.Client(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="openbmb/MiniCPM4.1-8B",
    messages=[
        {"role": "user", "content": "Write an article about Artificial Intelligence."},
    ],
    temperature=0.6,
    max_tokens=32768,
    extra_body=dict(add_special_tokens=True),  # Ensures special tokens are added for chat template
)

print(response.choices[0].message.content)

SGLang

Speculative Decoding

For accelerated inference with speculative decoding, follow these steps:

1. Download MiniCPM4.1 Draft Model

First, download the MiniCPM4.1 draft model:

cd /your_path
git clone https://huggingface.co/openbmb/MiniCPM4.1-8B-Eagle3

2. Install EAGLE3-Compatible SGLang

The EAGLE3 adaptation PR has been submitted. For now, use our repository for installation:

git clone https://github.com/LDLINGLINGLING/sglang.git
cd sglang
pip install -e .

3. Launch SGLang Server with Speculative Decoding

Start the SGLang server with speculative decoding enabled:

python -m sglang.launch_server \
  --model-path "openbmb/MiniCPM4.1-8B" \
  --host "127.0.0.1" \
  --port 30002 \
  --mem-fraction-static 0.9 \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path "your/path/MiniCPM4_1-8B-Eagle3-bf16" \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 32 \
  --temperature 0.7

4. Client Usage

The client usage remains the same for both standard and speculative decoding:

import openai

client = openai.Client(base_url=f"http://localhost:30002/v1", api_key="None")

response = client.chat.completions.create(
    model="openbmb/MiniCPM4.1-8B",
    messages=[
        {"role": "user", "content": "Write an article about Artificial Intelligence."},
    ],
    temperature=0.6,
    max_tokens=32768,
)

print(response.choices[0].message.content)

Note: Make sure to update the port number in the client code to match the server port (30002 in the speculative decoding example).

Configuration Parameters

• --speculative-algorithm EAGLE3: Enables EAGLE3 speculative decoding
• --speculative-draft-model-path: Path to the draft model for speculation
• --speculative-num-steps: Number of speculative steps (default: 3)
• --speculative-eagle-topk: Top-k parameter for EAGLE (default: 1)
• --speculative-num-draft-tokens: Number of draft tokens (default: 32)
• --mem-fraction-static: Memory fraction for static allocation (default: 0.9)

Standard Inference (Without Speculative Decoding)

For now, you need to install our forked version of SGLang.

git clone -b openbmb https://github.com/OpenBMB/sglang.git
cd sglang

pip install --upgrade pip
pip install -e "python[all]"

You can start the inference server by running the following command:

python -m sglang.launch_server --model openbmb/MiniCPM4.1-8B --trust-remote-code --port 30000 --chat-template chatml

Then you can use the chat interface by running the following command:

import openai

client = openai.Client(base_url=f"http://localhost:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="openbmb/MiniCPM4.1-8B",
    messages=[
        {"role": "user", "content": "Write an article about Artificial Intelligence."},
    ],
    temperature=0.6,
    max_tokens=32768,
)

print(response.choices[0].message.content)

CPM.cu

We recommend using CPM.cu for the inference of MiniCPM4 and MiniCPM4.1. CPM.cu is a CUDA inference framework developed by OpenBMB, which integrates efficient sparse, speculative sampling, and quantization techniques, fully leveraging the efficiency advantages of MiniCPM4 and MiniCPM4.1.

You can install CPM.cu by running the following command:

git clone https://github.com/OpenBMB/CPM.cu.git --recursive
cd CPM.cu
python3 setup.py install

You can run the following command to test the speed of the model.

python3 tests/long_prompt_gen.py # generate prompt.txt
python3 tests/test_generate.py --prompt-file prompt.txt

You can run the following command to infer with EAGLE3 speculative decoding algorithm.

python3 -m cpmcu.cli \
    --model-path $BASE_MODEL_PATH \
    --draft-model-path $EAGLE3_DRAFT_MODEL_PATH \
    --prompt-text "Tell me about Tsinghua University" \
    --use-eagle3 true

For more details about CPM.cu, please refer to the repo of CPM.cu.

llama.cpp and Ollama

We also support inference with llama.cpp and Ollama.

llama.cpp

You can download the GGUF format of MiniCPM4.1-8B model from huggingface and run it with llama.cpp for efficient CPU or GPU inference.

# case 1: main-cli
./build/bin/llama-cli -m MiniCPM4.1-8B-Q4_K_M.gguf -p "Write an article about Artificial Intelligence." -n 1500

# case 2: server
## launch server
./build/bin/llama-server -m MiniCPM4.1-8B-Q4_K_M.gguf --host 127.0.0.1 --port 8080 -c 4096 -fa on &

## send request
curl -X POST http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Write an article about Artificial Intelligence."}],
    "max_tokens": 1500
  }'

Ollama

Please refer to model hub for model download. After installing ollama package, you can use MiniCPM4.1 with following commands:

ollama run openbmb/minicpm4.1

Quantization (BitCPM4) and MiniCPM4 applications (Survey / MCP / Intel AIPC Client): see docs/README-legacy.md.

📄 LICENSE Copy Chapter

Model LICENSE

• This repository and MiniCPM models are released under the Apache-2.0 License.

Statement

• As a language model, MiniCPM generates content by learning from a vast amount of text.
• However, it does not possess the ability to comprehend or express personal opinions or value judgments.
• Any content generated by MiniCPM does not represent the viewpoints or positions of the model developers.
• Therefore, when using content generated by MiniCPM, users should take full responsibility for evaluating and verifying it on their own.

🏛 Institutions Copy Chapter

This project is developed by the following institutions:

• Modelbest Inc.
• THUNLP
• Gaoling School of Artificial Intelligence of RUC

📚 Citation Copy Chapter

• Please cite our paper: MiniCPM4 if you find our work valuable.

@article{minicpm4,
  title={Minicpm4: Ultra-efficient llms on end devices},
  author={MiniCPM, Team},
  journal={arXiv preprint arXiv:2506.07900},
  year={2025}
}

README Legacy Cn (`docs/README-legacy-cn.md`) Copy Section

MiniCPM 历史专题

早期 MiniCPM 模型的附属说明。当前主推模型见 主 README。

English: README-legacy.md

BitCPM4: 模型量化 Copy Chapter

BitCPM4 是基于 MiniCPM 系列模型进行量化感知训练（QAT）后得到的三值量化模型，在训练效率和模型参数效率实现了有效的提升。

• 训练方法改进
  • 在小规模模型上进行风洞实验，搜索训练所需的训练超参。
  • 通过使用一阶段高精训练+二阶段 QAT 的方法，充分利用已经完成或部分完成训练的高精度模型，极大地压缩了 QAT 阶段所需要的算力。
• 高效参数效率
  • 模型使用 1.58Bit 的位宽达到的性能对标与同参数量级别的全精度模型，模型参数效率高。

BitCPM4 评测

BitCPM4 在测试中的表现可以对标同级别的业界主流全精度模型。

BitCPM4 模型推理

BitCPM4 开源的模型参数为伪量化形式，可以直接使用 Huggingface 框架进行推理。

MiniCPM4 应用 Copy Chapter

查看 MiniCPM4 的应用

MiniCPM4-Survey: 综述生成

MiniCPM4-Survey 是由 THUNLP、中国人民大学和 ModelBest 联合开发的开源大语言模型智能体。它基于 MiniCPM4-8B 基座模型，接受用户质量作为输入，自主生成可信的长篇综述论文。
主要特性包括：

• 计划-检索-写作生成框架 — 我们提出了一个多智能体生成框架，包含三个核心阶段：计划（定义综述的整体结构）、检索（生成合适的检索关键词）和写作（利用检索到的信息，生成连贯的段落）。
• 高质量数据集构建——我们收集并处理大量人类专家写作的综述论文，构建高质量训练集。同时，我们收集大量研究论文，构建检索数据库。
• 多方面奖励设计 — 我们精心设计了包含结构、内容和引用的奖励，用于评估综述的质量，在强化学习训练阶段作奖励函数。
• 多步强化学习训练策略 — 我们提出了一个上下文管理器，以确保在促进有效推理的同时保留必要的信息，并构建了并行环境，维持强化学习训练高效。

使用与演示案例

详见此处

评估

GPT-4o 对综述生成系统的性能比较。“G2FT” 代表 Gemini-2.0-Flash-Thinking，“WTR1-7B” 代表 Webthinker-R1-7B。由于 Webthinker 不包括引用功能，OpenAI Deep Research 在导出结果时不提供引用，因此省略了对它们的 FactScore 评估。我们的技术报告中包含评测的详细信息。

MiniCPM4-MCP: MCP增强的工具调用

MiniCPM4-MCP 是由清华大学自然语言处理实验室（THUNLP）、中国人民大学与 ModelBest 联合开发的开源本地大语言模型代理，它基于 MiniCPM-4-8B，拥有 80 亿参数。它能够通过 MCP 协议与各种工具和数据资源交互，解决多种真实世界任务。截至目前，MiniCPM4-MCP 已支持：

• 涵盖 16 个 MCP 服务器（servers）中工具的使用：这些服务器所包含的工具横跨了办公类、生活类、通讯类、资讯类、工作管理类等.

• 单工具使用的能力：可使用符合 MCP 协议的工具进行单一工具的一步或多步调用。

• 跨工具组合使用的能力：可组合使用符合 MCP 协议的不同工具。

使用与演示案例

详见此处

评估

MiniCPM Intel AIPC Client: 端侧大模型客户端

MiniCPM Intel AIPC Client 是面壁智能和 Intel 合作推出的端侧大模型客户端，专为搭载 Intel Core Ultra 系列处理器的设备设计，旨在为开发者、研究人员与 AI 爱好者带来低延迟、高效率、高隐私的本地大模型使用体验。其核心特性如下：

• 深度适配 Intel 硬件：全面支持 Intel Core Ultra 系列处理器，实现与硬件的深度融合，充分释放硬件性能，让用户无需依赖云端，在本地设备上就能流畅运行大模型。
• 基于 OpenVINO 的极致优化：基于 OpenVINO 推理框架进行深度优化，大幅提升推理效率，推理速度最高可达每秒 80 tokens，确保模型响应迅速，无论是快速问答还是复杂任务处理，都能高效完成。
• 隐私安全保障：采用本地部署方式，所有数据处理均在本地设备完成，避免数据上传至云端带来的隐私风险，让用户使用更安心，尤其适合对数据隐私要求较高的场景。
• 面向多元用户群体：无论是追求前沿技术的开发者，专注学术研究的科研人员，还是热衷于探索 AI 应用的爱好者，都能通过 MiniCPM Intel AIPC Client，轻松体验本地大模型的强大功能，开启个性化的 AI 探索之旅 。

配置要求：

• 建议使用英特尔酷睿 ultra7 及以上移动端处理器
• 建议运行内存 32GB 及以上

应用下载：

下载地址

README Legacy (`docs/README-legacy.md`) Copy Section

MiniCPM Legacy Topics

Auxiliary topics for earlier MiniCPM releases. For the current flagship model, see the main README.

中文版: README-legacy-cn.md

BitCPM4: Quantization Copy Chapter

BitCPM4 are ternary quantized models derived from the MiniCPM series models through quantization-aware training (QAT), achieving significant improvements in both training efficiency and model parameter efficiency.

• Improvements of the training method
  • Searching hyperparameters with a wind-tunnel on a small model.
  • Using a two-stage training method: training in high-precision first and then QAT, making the best of the trained high-precision models and significantly reducing the computational resources required for the QAT phase.
• High parameter efficiency
  • Achieving comparable performance to full-precision models of similar parameter models with a bit width of only 1.58 bits, demonstrating high parameter efficiency.

BitCPM4 Evaluation

BitCPM4's performance is comparable with other full-precision models in same model size.

BitCPM4 Inference

BitCPM4's parameters are stored in a fake-quantized format, which supports direct inference within the Huggingface framework.

MiniCPM4 Application Copy Chapter

Click to view details about MiniCPM4 Application

MiniCPM4-Survey: Trustworthy Survey Generation

MiniCPM4-Survey is an open-source LLM agent model jointly developed by THUNLP, Renmin University of China and ModelBest. Built on MiniCPM4-8B, it accepts users' quiries as input and autonomously generate trustworthy, long-form survey papers.

Key features include:

• Plan-Retrieve-Write Survey Generation Framework — We propose a multi-agent generation framework, which operates through three core stages: planning (defining the overall structure of the survey), retrieval (generating appropriate retrieval keywords), and writing (synthesizing the retrieved information to generate coherent section-level content).

• High-Quality Dataset Construction — We gather and process lots of expert-written survey papers to construct a high-quality training dataset. Meanwhile, we collect a large number of research papers to build a retrieval database.

• Multi-Aspect Reward Design — We carefully design a reward system with three aspects (structure, content, and citations) to evaluate the quality of the surveys, which is used as the reward function in the RL training stage.

• Multi-Step RL Training Strategy — We propose a Context Manager to ensure retention of essential information while facilitating efficient reasoning, and we construct Parallel Environment to maintain efficient RL training cycles.

Demo and Quick Start

See here

Performance Evaluation

Performance comparison of the survey generation systems. "G2FT" stands for Gemini-2.0-Flash-Thinking, and "WTR1-7B" denotes Webthinker-R1-7B. FactScore evaluation was omitted for Webthinker, as it does not include citation functionality, and for OpenAI Deep Research, which does not provide citations when exporting the results.

MiniCPM4-MCP: Tool Use with Model Context Protocol

MiniCPM4-MCP is an open-source on-device LLM agent model jointly developed by THUNLP, Renmin University of China and ModelBest, built on MiniCPM-4 with 8 billion parameters. It is capable of solving a wide range of real-world tasks by interacting with various tool and data resources through MCP. As of now, MiniCPM4-MCP supports the following:

• Utilization of tools across 16 MCP servers: These servers span various categories, including office, lifestyle, communication, information, and work management.

• Single-tool-calling capability: It can perform single- or multi-step tool calls using a single tool that complies with the MCP.

• Cross-tool-calling capability: It can perform single- or multi-step tool calls using different tools that complies with the MCP.

Demo

Demo is available in this link.

Performance Evaluation

MiniCPM Intel AIPC Client: A New Edge Large Model Powerhouse

Developed in collaboration between Mianbi Intelligence and Intel, the MiniCPM Intel AIPC Client is an edge large model client specially designed for devices equipped with Intel Core Ultra series processors. It delivers a low-latency, high-efficiency, and privacy-preserving local large model experience for developers, researchers, and AI enthusiasts. Its core features include:

Key Features

• Deep Intel Hardware Adaptation
  Fully compatible with Intel Core Ultra series processors, enabling deep integration with hardware to unleash peak performance. Users can run large models smoothly on local devices without relying on cloud services.

• Extreme Optimization Based on OpenVINO
  Deeply optimized with the OpenVINO inference framework, it significantly boosts inference efficiency, reaching up to 80 tokens per second. This ensures rapid model response for both quick queries and complex task processing.

• Privacy and Security Assurance
  Adopting local deployment, all data processing is completed on the device, eliminating privacy risks from cloud uploads. This provides users with peace of mind, especially for scenarios with high data privacy requirements.

• Catering to Diverse User Groups
  Whether for developers chasing cutting-edge technologies, researchers focused on academic studies, or enthusiasts eager to explore AI applications, the MiniCPM Intel AIPC Client enables easy access to the power of local large models, opening the door to personalized AI exploration.

System Requirements

• Recommended processor: Intel Core Ultra 7 or higher (mobile version)
• Recommended RAM: 32GB or above

Download

download

README Minicpm1 Cn (`docs/README-minicpm1-cn.md`) Copy Section

text Copy

**中文** | [English](README-minicpm1-en.md)

MiniCPM 1.0

MiniCPM-2B 语言模型有 24亿（2.4B）的非词嵌入参数量, 总计 2.7B 参数量。

• 经过 SFT 后，MiniCPM-2B 在公开评测集上与 Mistral-7B 表现相近（中文、数学、代码能力更优），整体性能超越 Llama2-13B、MPT-30B、Falcon-40B 等模型。
• 经过 DPO 后，MiniCPM-2B 在 MTBench 上也超越了 Llama2-70B-Chat、Vicuna-33B、Mistral-7B-Instruct-v0.1、Zephyr-7B-alpha 等众多代表性开源大模型。

注意：为了保证在学术研究用途上模型的通用性，我们未对 MiniCPM-2B 进行任何身份认同训练。同时由于我们用 ShareGPT 开源语料作为部分训练数据，模型可能会输出类似 GPT 系列模型的身份认同信息。

评测结果 Copy Chapter

评测设置 Copy Section

• 由于大模型评测难以统一，且大量评测也没有公开的prompt和测试代码，对于具体评测方式，我们只能尽量做到适合各类模型。
• 整体而言，我们测试时采用统一的prompt输入，并按照各模型对应的模板进行输入调整。
• 评测脚本及prompt已开源在我们的Github仓库中，也欢迎更多开发者来不断改进我们的评测方式。
  • 文本评测部分，采用了我们的开源大模型能力评测框架UltraEval。以下为开源模型复现流程：
    • 安装UltraEval
      shell Copy

      git clone https://github.com/OpenBMB/UltraEval.git
      cd UltraEval
      pip install -e .

    • 下载相关数据并解压处理
      shell Copy

      wget -O RawData.zip "https://cloud.tsinghua.edu.cn/f/71b5232264ae4833a4d0/?dl=1"
      unzip RawData.zip
      python data_process.py

    • 执行评测脚本(提供了模板，可自定义)
      shell Copy

      bash run_eval.sh

部署模式 Copy Section

• 因为MiniCPM采用Mup的结构，与现有模型在具体计算上有细微差别，我们是基于vllm=0.2.2版本进行了我们模型的实现。
• 对于非MiniCPM模型，我们采用了vllm=0.2.7的最新版本进行推理。

评测度量 Copy Section

• 对于QA任务（选择题任务），我们选用两种方式进行测试：
  • PPL：将选项作为题目生成的延续，并根据各个选项的PPL来进行答案选择；
  • 第二种是直接生成答案选项。
• 对于不同模型，这两种方式得到的结果差异较大。MiniCPM两种模式上的结果较为接近，而Mistral-7B-v0.1等模型在PPL上表现较好，直接生成上效果较差。
• 在具体评测时，我们以两种评测方式得分的最高者为最终结果，以此保证对比的公平性(以下表格中*号表示采用PPL)。

文本模型评测 Copy Section

越级比较:

同级比较：

Chat模型比较：

DPO后模型比较：

快速上手 Copy Chapter

在线体验 Copy Section

• Colab

基于Gradio的网页版Demo Copy Section

• 使用如下命令启动基于Gradio的网页版demo：

# generation powered by vllm
python demo/minicpm/vllm_based_demo.py --model_path <vllmcpm_repo_path>
# generation powered by huggingface
python demo/minicpm/hf_based_demo.py --model_path <hf_repo_path>

HuggingFace 推理 Copy Section

MiniCPM-2B

安装transformers>=4.36.0以及accelerate后，运行以下代码：

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
torch.manual_seed(0)

path = 'openbmb/MiniCPM-2B-dpo-bf16'
tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map='cuda', trust_remote_code=True)

responds, history = model.chat(tokenizer, "山东省最高的山是哪座山, 它比黄山高还是矮？差距多少？", temperature=0.5, top_p=0.8, repetition_penalty=1.02)
print(responds)

MiniCPM-2B （Llama Format）

我们将MiniCPM的模型权重转化成了Llama代码可以直接调用的格式，以便大家尝试:

import torch
from transformers import LlamaTokenizerFast, LlamaForCausalLM
model_path = "openbmb/MiniCPM-2B-dpo-bf16-llama-format"
tokenizer = LlamaTokenizerFast.from_pretrained(model_path)
model = LlamaForCausalLM.from_pretrained(model_path, torch_dtype=torch.bfloat16, device_map='cuda', trust_remote_code=True)

prompt="Now you act like a terminal situated within a beginner's C++ practice repository folder, please provide the output for the command: `ls -l`"
input_ids = tokenizer.encode("<用户>{}<AI>".format(prompt), return_tensors='pt', add_special_tokens=True).cuda()
responds = model.generate(input_ids, temperature=0.3, top_p=0.8, repetition_penalty=1.02, max_length=1024)
responds = tokenizer.decode(responds[0], skip_special_tokens=True)
print(responds)

vLLM 推理 Copy Section

安装 vLLM。

pip install "vllm>=0.4.1"

具体推理代码见这里。

SGLang 推理 Copy Section

安装 SGLang。

• 首先需要启动一个服务:

python -m sglang.launch_server --model-path openbmb/MiniCPM-2B-dpo-fp16 --trust-remote-code --port 30000

• 下面是一个推理代码的样例:

from sglang import function, gen, set_default_backend, RuntimeEndpoint

@function
def text_qa(s, question):
    s += "<用户>" + question + "<AI>"
    s += gen("answer", max_tokens=1024, temperature=0.7, top_p=0.7)

set_default_backend(RuntimeEndpoint("http://localhost:30000"))

state = text_qa.run(
    question="What is the capital of China?",
)

print(state["answer"])

llama.cpp、Ollama、fastllm、mlx_lm推理 Copy Section

MiniCPM支持llama.cpp 、ollama、fastllm、mlx_lm推理。感谢@runfuture对llama.cpp和ollama的适配。

请参考 MiniCPM 知识库中的边端部署教程。

模型量化 Copy Section

请参考 MiniCPM 知识库中的量化指南。

模型微调 Copy Section

• 一张 1080/2080 可实现高效参数微调：代码
• mlx 微调：教程
• xtuner: MiniCPM高效率微调的不二选择
• LLaMA-Factory：MiniCPM微调一键式解决方案

README Minicpm1 En (`docs/README-minicpm1-en.md`) Copy Section

text Copy

[中文](README-minicpm1-cn.md) | **English**

MiniCPM 1.0

Introduction Copy Chapter

MiniCPM-2B is a dense language model with only 2.4B parameters excluding embeddings (2.7B in total).

• MiniCPM has very close performance compared with Mistral-7B on open-sourced general benchmarks with better ability on Chinese, Mathematics and Coding after SFT. The overall performance exceeds Llama2-13B, MPT-30B, Falcon-40B, etc.

• After DPO, MiniCPM outperforms Llama2-70B-Chat, Vicuna-33B, Mistral-7B-Instruct-v0.1, Zephyr-7B-alpha, etc. on MTBench.

Note: To ensure the generality of the model for academic research purposes, we have not subject it to any identity-specific training. Meanwhile, as we use ShareGPT open-source corpus as part of the training data, the model may output identity-related information similar to the GPT series models.

Evaluation Results Copy Chapter

Evaluation Settings Copy Section

• Since it is difficult to standardize the evaluation of LLMs and there is no public prompt and test code for a large number of evaluations, we can only try our best to make it suitable for all types of models in terms of specific evaluation methods.
• Overall, we use a unified prompt input for testing, and adjust the input according to the corresponding template for each model.
• The evaluation scripts and prompts have been open-sourced in our Github repository, and we welcome more developers to continuously improve our evaluation methods.
  • For the text evaluation part, we use our open source large model capability evaluation framework UltraEval. The following is the open source model reproduction process:
    • install UltraEval
      shell Copy

      git clone https://github.com/OpenBMB/UltraEval.git
      cd UltraEval
      pip install -e .

    • Download the relevant data and unzip it for processing
      shell Copy

      wget -O RawData.zip "https://cloud.tsinghua.edu.cn/f/71b5232264ae4833a4d0/?dl=1"
      unzip RawData.zip
      python data_process.py

    • Execute evaluation scripts (templates are provided and can be customized)
      shell Copy

      bash run_eval.sh

Deployment mode Copy Section

• Because MiniCPM uses the structure of Mup, which is slightly different from existing models in terms of specific computations, we have based the implementation of our model on the vllm=0.2.2 version.
• For non-MiniCPM models, we directly sampled the latest version of vllm=0.2.7 for inference.

Evaluation method Copy Section

• For the QA task (multiple-choice task), we chose to test in two ways:
  • PPL: The options are used as a continuation of the question generation and the answer selection is based on the PPL of each option;
  • The second is to generate the answer options directly.
• For different models, the results obtained by these two approaches vary widely. the results on both MiniCPM models are closer, while models such as Mistral-7B-v0.1 perform better on PPL and worse on direct generation.
• In the specific evaluation, we take the higher score of the two evaluation methods as the final result, so as to ensure the fairness of the comparison (* in the following table indicates the PPL).

Text evaluation Copy Section

DPO evaluation Copy Section

Quick Start Copy Chapter

Online Copy Section

• Colab

Web-demo based on Gradio Copy Section

Using the following command can launch the gradio-based demo.

# generation powered by vllm
python demo/minicpm/vllm_based_demo.py --model_path <vllmcpm_repo_path>
# generation powered by huggingface
python demo/minicpm/hf_based_demo.py --model_path <hf_repo_path>

Huggingface Inference Copy Section

MiniCPM-2B

Install transformers>=4.36.0 and accelerate，run the following python code:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
torch.manual_seed(0)

path = 'openbmb/MiniCPM-2B-dpo-bf16'
tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map='cuda', trust_remote_code=True)

responds, history = model.chat(tokenizer, "Which city is the capital of China?", temperature=0.8, top_p=0.8)
print(responds)

MiniCPM-2B (Llama Format)

To facilitate ease of use, we have converted the model weights of MiniCPM to adapt to the structure of the LLaMA model:

import torch
from transformers import LlamaTokenizerFast, LlamaForCausalLM
model_path = "openbmb/MiniCPM-2B-dpo-bf16-llama-format"
tokenizer = LlamaTokenizerFast.from_pretrained(model_path)
model = LlamaForCausalLM.from_pretrained(model_path, torch_dtype=torch.bfloat16, device_map='cuda', trust_remote_code=True)

prompt="Now you act like a terminal situated within a beginner's C++ practice repository folder, please provide the output for the command: `ls -l`"
input_ids = tokenizer.encode("<User>{}<AI>".format(prompt), return_tensors='pt', add_special_tokens=True).cuda()
responses = model.generate(input_ids, temperature=0.3, top_p=0.8, repetition_penalty=1.02, max_length=1024)
responses = tokenizer.decode(responses[0], skip_special_tokens=True)
print(responses)

vLLM Inference Copy Section

Install vLLM.

pip install "vllm>=0.4.1"

See here for the inference code.

SGLang Inference Copy Section

Install SGLang.

• First, launch a server:

python -m sglang.launch_server --model-path openbmb/MiniCPM-2B-dpo-fp16 --trust-remote-code --port 30000

• You can use it for inference as shown below:

from sglang import function, gen, set_default_backend, RuntimeEndpoint

@function
def text_qa(s, question):
    s += "<User>" + question + "<AI>"
    s += gen("answer", max_tokens=1024, temperature=0.7, top_p=0.7)

set_default_backend(RuntimeEndpoint("http://localhost:30000"))

state = text_qa.run(
    question="What is the capital of China?",
)

print(state["answer"])

llama.cpp, Ollama, fastllm, mlx_lm Inference Copy Section

We have supported inference with llama.cpp, ollama, fastllm, mlx_lm. Thanks to @runfuture for the adaptation of llama.cpp and ollama.

Please refer to Edge Deployment Tutorial.

Quantization Copy Section

Please refer to Quantization Tutorial.

Fine-Tuning Copy Section

• With parameter-efficient tuning, we can tune MiniCPM using one piece of NVIDIA GeForce GTX 1080/2080: code.
• mlx finetune: Guideline

• xtuner: The best choice to do parameter-efficient tuning on MiniCPM
• LLaMA-Factory：One click solution of finetuning MiniCPM

README Minicpm2 Cn (`docs/README-minicpm2-cn.md`) Copy Section

text Copy

**中文** | [English](README-minicpm2-en.md)

MiniCPM 2.0

MiniCPM 2.0 系列模型对 MiniCPM 进行了多个维度的升级，包括以下模型版本：

• MiniCPM-2B-128k：将 MiniCPM-2B 的上下文长度从 4k 扩展至 128k，在 InfiniteBench 测试集上优于 ChatGLM3-6B-128k、Yi-6B-200k 等更大参数量的模型。
• MiniCPM-MoE-8x2B：基于 MiniCPM-2B 进行 MoE 扩展，综合表现相比于 MiniCPM-2B 平均提高 4.5 个百分点。
• MiniCPM-1B：相比于 MiniCPM-2B 成本下降 60%，综合表现仍然优于 LLaMA2-13B。
• MiniCPM-S-1B：在保持下游任务性能无损的前提下，FFN 层实现了 87.89% 的平均稀疏度，将 FFN FLOPs 降低了 84%。结合 PowerInfer 推理框架，解码速度提升约 2.8 倍。

评测结果 Copy Chapter

MiniCPM-2B-128k 模型评测 Copy Section

MiniCPM-MoE-8x2B 模型评测 Copy Section

注：* 表示结果取自技术报告。† 表示评测集为MBPP全集。

MiniCPM-S-1B 评测结果 Copy Section

• 代码生成：在 HumanEval（0-shot）和 MBPP（3-shot）上的平均 pass@1 得分。
• 常识推理：在 PIQA、SIQA、HellaSwag、WinoGrande 和 COPA 上的平均 0-shot 准确率。
• 阅读理解：在 BoolQ、LAMBADA 和 TyDi QA 上的平均 0-shot 准确率。

其他测试集：我们报告在GSM8K（8-shot）、MMLU（5-shot）、BBH（3-shot）和 AGI-Eval（0-shot）上的平均准确率。

注：

1. ReluLLaMA-7B 和 ReluLLaMA-13B 的下载链接分别是 7B and 13B。"ProSparse-7B*"、"ProSparse-13B*" 和 "MiniCPM-S-1B*" 代表没有激活阈值偏移的 ProSparse 版本。
2. 对于 PIQA、SIQA、HellaSwag、WinoGrande、COPA、BoolQ、LAMBADA、TyDi QA 和 AGI-Eval，我们根据各个选项的 PPL 来进行答案选择。对于 GSM8K、MMLU 和 BBH，我们直接生成答案。

模型推理 Copy Chapter

HuggingFace、vLLM推理 Copy Section

参考 MiniCPM 1.0 中的模型推理部分。

Powerinfer 推理 Copy Section

针对 MiniCPM-S-1B 模型，我们可以使用 Powerinfer 进行推理加速，使用方法如下：

1. 保证cmake版本3.17以上，如果已经安装过，则跳过此步骤

# 下载安装包
  sudo wget https://cmake.org/files/v3.23/cmake-3.23.0.tar.gz
  # 解压安装包
  sudo tar -zxvf cmake-3.23.0.tar.gz
  # 配置安装环境
  sudo ./configure
  sudo make -j8
  # 编译安装
  sudo make install
  # 查看安装后版本
  cmake --version
  # 返回版本号则安装成功
  #cmake version 3.23.0

2. 安装powerinfer：

git clone https://github.com/SJTU-IPADS/PowerInfer
  cd PowerInfer
  pip install -r requirements.txt # install Python helpers' dependencies

3. cpu版本powerinfer编译,如果你的机器只有cpu，或者只想使用cpu进行推理，则运行以下命令：

cmake -S . -B build
  cmake --build build --config Release

4. gpu版本powerinfer编译,如果你的机器有gpu，则可以运行以下命令：

cmake -S . -B build -DLLAMA_CUBLAS=ON
  cmake --build build --config Release

5. 获取稀疏模型

git clone https://huggingface.co/openbmb/MiniCPM-S-1B-sft-gguf/tree/main
#or
git clone https://modelscope.cn/models/OpenBMB/MiniCPM-S-1B-sft-gguf

6. 模型推理：

cd PowerInfer
# 以下是命令模版，output_token_count为最大输出tokens，thread_num 为线程数，prompt为输入prompt字符
#./build/bin/main -m /PATH/TO/MODEL -n $output_token_count -t $thread_num -p $prompt
# 以下是示例
./build/bin/main -m /root/ld/ld_model_pretrain/1b-s-minicpm/MiniCPM-S-1B-sft.gguf -n 2048 -t 8 -p '<用户>hello,tell me a story please.<AI>'

README Minicpm2 En (`docs/README-minicpm2-en.md`) Copy Section

text Copy

[中文](README-minicpm2-cn.md) | **English**

MiniCPM 2.0

Introduction Copy Chapter

MiniCPM 2.0 series upgrade MiniCPM in multiple dimensions, including:

• MiniCPM-2B-128k：Extend the length of MiniCPM-2B context window to 128k, outperform larger models such as ChatGLM3-6B-128k、Yi-6B-200k on InfiniteBench.
• MiniCPM-MoE-8x2B：Upcycling from MiniCPM-2B. Compared to MiniCPM-2B, the overall performance improves by an average of 4.5pp.
• MiniCPM-1B: 60% inference cost reduction compared with MiniCPM-2B, while still showing better overall performance than LLaMA2-13B.
• MiniCPM-S-1B: The FFN layer achieves an average sparsity of 87.89% and reduces FFN FLOPs by 84%, while maintaining no performance loss in downstream tasks. Combined with the PowerInfer, MiniCPM-S-1B inferece speed increase is approximately 2.8x.

Evaluation Results Copy Chapter

MiniCPM-2B-128k Copy Section

MiniCPM-MoE-8x2B Copy Section

Note：* means evaluation results are directly taken from their technical reports. † means evaluation results on the full set of MBPP, instead of the hand-verified set.

MiniCPM-S-1B Copy Section

• Code Generation：Average pass@1 score of HumanEval(0-shot) and MBPP(3-shot).
• Commonsense Reasoning: Average 0-shot accuracy of PIQA, SIQA, HellaSwag, WinoGrande and COPA.
• Reading Comprehension: Average 0-shot accuracy of BoolQ, LAMBADA and TyDi-QA.
• Other Benchmarks: We report average performance of GSM8K(8-shot)、MMLU(5-shot)、BBH(3-shot) and AGI-Eval(0-shot).

Note：

1. ReluLLaMA-7B and ReluLLaMA-13B. "ProSparse-7B*"、"ProSparse-13B*" and "MiniCPM-S-1B*" represent ProSparse versions that don't have activation thresholds offset.
2. For PIQA, SIQA, HellaSwag, WinoGrande, COPA, BoolQ, LAMBADA, TyDi QA and AGI-Eval, we adopt ppl-based evaluation. For GSM8K, MMLU and BBH, we perform generation-based evaluation.

Inference Copy Chapter

HuggingFace, vLLM Copy Section

Please refer to Inference section in MiniCPM1.0.

PowerInfer Copy Section

Currently, PowerInfer is exclusively tailored for the MiniCPM-S-1B model; support for other versions is not yet available, stay tuned.

1. Ensure your cmake version is 3.17 or above. If you have already installed it, you can skip this step.

# Download the installation package
    sudo wget https://cmake.org/files/v3.23/cmake-3.23.0.tar.gz
    # Extract the installation package
    sudo tar -zxvf cmake-3.23.0.tar.gz
    # Configure the installation environment
    sudo ./configure
    sudo make -j8
    # Compile and install
    sudo make install
    # Check the version after installation
    cmake --version
    # If the version number is returned, the installation was successful
    # cmake version 3.23.0

2. Install PowerInfer:

git clone https://github.com/SJTU-IPADS/PowerInfer
  cd PowerInfer
  pip install -r requirements.txt # install Python helpers' dependencies

3. Compile the CPU version of PowerInfer. If your machine only has a CPU, or if you want to perform inference using the CPU, run the following commands:

cmake -S . -B build
  cmake --build build --config Release

4. Compile the GPU version of PowerInfer. If your machine has a GPU, you can run the following commands:

cmake -S . -B build -DLLAMA_CUBLAS=ON
  cmake --build build --config Release

5. Retrieve the sparse model:

git clone https://huggingface.co/openbmb/MiniCPM-S-1B-sft-gguf/tree/main
#or
git clone https://modelscope.cn/models/OpenBMB/MiniCPM-S-1B-sft-gguf

6. Model Inference:

cd PowerInfer
# Below is the command template. output_token_count refers to the maximum output tokens, thread_num is the number of threads, and prompt is the input prompt text.
#./build/bin/main -m /PATH/TO/MODEL -n $output_token_count -t $thread_num -p $prompt
# Below is an example
./build/bin/main -m /root/ld/ld_model_pretrain/1b-s-minicpm/MiniCPM-S-1B-sft.gguf -n 2048 -t 8 -p '<User>hello,tell me a story please.<AI>'

README Minicpm3 Cn (`docs/README-minicpm3-cn.md`) Copy Section

text Copy

**中文** | [English](README-minicpm3-en.md)

MiniCPM 3.0

MiniCPM 3.0 是一个 4B 参数量的语言模型，相比 MiniCPM1.0/2.0，功能更加全面，综合能力大幅提升，多数评测集上的效果比肩甚至超越众多 7B-9B 模型。

• 支持工具调用🛠️（Function Calling）和代码解释器💻（Code Interpreter）：Berkeley Function Calling Leaderboard (BFCL) 上取得 9B 规模以下 SOTA，超越 GLM-4-9B-Chat、Qwen2-7B-Instruct。
• 超强的推理能力🧮：数学能力方面，MathBench 上的效果超越 GPT-3.5-Turbo 以及多个 7B-9B 模型。在非常具有挑战性的 LiveCodeBench 上，效果超越 Llama3.1-8B-Instruct。
• 出色的中英文指令遵循能力🤖：英文指令遵循 IFEval、中文指令遵循 FollowBench-zh 效果超越 GLM-4-9B-Chat、Qwen2-7B-Instruct。
• 长文本能力：原生支持 32k 上下文长度，32k 长度内大海捞针全绿。提出 LLMxMapReduce ，理论可处理的上下文长度达到 +∞，在综合性长文本评测基准 InfiniteBench 平均得分超越GPT-4、KimiChat等标杆模型。
• RAG能力：我们发布了 MiniCPM RAG 套件。基于 MiniCPM 系列模型的 MiniCPM-Embedding、MiniCPM-Reranker 在中文、中英跨语言检索测试中取得 SOTA 表现；针对 RAG 场景的 MiniCPM3-RAG-LoRA 在开放域问答等多项任务上超越 Llama3-8B、Baichuan2-13B 等模型。

评测结果 Copy Chapter

综合评测 Copy Section

工具调用能力 Copy Section

我们在 Berkeley Function Calling Leaderboard (BFCL) 上测试了模型的工具调用能力，MiniCPM3-4B 在该榜单上的表现超越了多个 7B-9B 参数量的模型，优于 GPT-3.5-Turbo-0125。

长文本能力 Copy Section

在 32k 的上下文长度进行大海捞针测试，结果如下图：

同时我们提出LLMxMapReduce，利用分治的策略，理论上可以处理无限长度的文本。我们在InfiniteBench上测试了模型的长文本处理能力，在LLMxMapReduce框架的加持下，MiniCPM3-4B在这个榜单的平均得分能够超越 GPT-4、KimiChat 等标杆模型。

模型推理 Copy Chapter

Huggingface Copy Section

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
torch.manual_seed(0)

path = 'openbmb/MiniCPM3-4B'
tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map='cuda', trust_remote_code=True)

responds, history = model.chat(tokenizer, "请写一篇关于人工智能的文章，详细介绍人工智能的未来发展和隐患。", temperature=0.7, top_p=0.7)
print(responds)

SGLang（推荐） Copy Section

• 安装

参考 SGLang 官方仓库，通过源码安装最新版本。

• 启动推理服务

python -m sglang.launch_server --model openbmb/MiniCPM3-4B --trust-remote-code --port 30000 --chat-template chatml

• 使用示例

from sglang import function, system, user, assistant, gen, set_default_backend, RuntimeEndpoint

@function
def multi_turn_question(s, question_1, question_2):
    s += user(question_1)
    s += assistant(gen("answer_1", max_tokens=1024))
    s += user(question_2)
    s += assistant(gen("answer_2", max_tokens=1024))

set_default_backend(RuntimeEndpoint("http://localhost:30000"))

state = multi_turn_question.run(
    question_1="介绍一下人工智能",
    question_2="写一篇关于它的文章",
)

for m in state.messages():
    print(m["role"], ":", m["content"])

vLLM Copy Section

• 安装 vllm
  shell Copy

  pip install "vllm>=0.6.2"

• 推理
  python Copy

  from transformers import AutoTokenizer
  from vllm import LLM, SamplingParams
  
  model_name = "openbmb/MiniCPM3-4B"
  prompt = [{"role": "user", "content": "请写一篇关于人工智能的文章，详细介绍人工智能的未来发展和隐患。"}]
  
  tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
  input_text = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True)
  
  llm = LLM(model=model_name,
      trust_remote_code=True,
      tensor_parallel_size=1
  )
  sampling_params = SamplingParams(top_p=0.7, temperature=0.7, max_tokens=1024)
  
  outputs = llm.generate(prompts=input_text, sampling_params=sampling_params)
  
  print(outputs[0].outputs[0].text)

llama.cpp Copy Section

我们提供了 MiniCPM3 的 GGUF 版本，可以直接使用 llama.cpp 推理。

• 安装 llama.cpp
  shell Copy

  git clone https://github.com/ggerganov/llama.cpp
    cd llama.cpp
    make

• 推理
  shell Copy

  ./llama-cli -c 1024 -m minicpm3-4b-fp16.gguf -n 1024 --top-p 0.7 --temp 0.7 --prompt "<|im_start|>user\n请写一篇关于人工智能的文章，详细介绍人工智能的未来发展和隐患。<|im_end|>\n<|im_start|>assistant\n"

模型微调 Copy Chapter

LLaMA-Factory Copy Section

目前模型微调支持 LLaMA-Factory，使用方法参考 LLaMA-Factory 微调。

进阶功能 Copy Chapter

对于以下进阶功能，我们的样例代码中使用 vLLM 进行推理。

工具调用 Copy Section

我们提供了使用 MiniCPM3 调用工具的示例代码：

cd demo/minicpm3/function_call
python function_call.py

如果你想启动一个能够调用工具的推理服务，使用以下代码：

cd demo/minicpm3/function_call
pip install -r requirements.txt
python openai_api_server.py \
    --model openbmb/MiniCPM3-4B \
    --served-model-name MiniCPM3-4B \
    --chat-template chatml.jinja \
    --dtype auto \
    --api-key token-abc123 \
    --tensor-parallel-size 1 \
    --trust-remote-code

下面是一个调用搜索工具回答问题的演示：

代码解释器 Copy Section

我们提供了一个 MiniCPM3 使用代码解释器的示例代码：

cd demo/minicpm3/code_interpreter
pip install -r requirements.txt
python code_interpreter.py openbmb/MiniCPM3-4B

下面是一个使用代码解释器生成二维码的演示：

README Minicpm3 En (`docs/README-minicpm3-en.md`) Copy Section

text Copy

[中文](README-minicpm3-cn.md) | **English**

MiniCPM 3.0

MiniCPM 3.0 is a language model with 4 billion parameters. Compared to MiniCPM 1.0/2.0, it offers more comprehensive features and a significant improvement in overall capabilities. Its performance on most evaluation benchmarks rivals or even surpasses many models with 7B-9B parameters.

• Supports Function Call🛠️ and Code Interpreter💻: Achieved SOTA among models with fewer than 9B parameters on the Berkeley Function Calling Leaderboard (BFCL), outperforming GLM-4-9B-Chat and Qwen2-7B-Instruct.
• Exceptional Reasoning Ability🧮: In terms of math abilities, it outperforms GPT-3.5-Turbo and several 7B-9B models on MathBench. On the highly challenging LiveCodeBench, it surpasses Llama3.1-8B-Instruct.
• Outstanding Instruction-Following in English and Chinese🤖: Exceeds GLM-4-9B-Chat and Qwen2-7B-Instruct on English instruction following with IFEval and on Chinese instruction following with FollowBench-zh.
• Long Context Capability: Natively supports 32k context length, with flawless performance. We introduce the LLMxMapReduce framework, theoretically enabling processing of context lengths up to infinity. Enhanced by LLMxMapReduce, MiniCPM3-4B achieves performance comparable to GPT-4 and KimiChat on InfiniteBench.
• RAG Capability：We release MiniCPM RAG Suite. Based on the MiniCPM series models, MiniCPM-Embedding and MiniCPM-Reranker achieve SOTA performance on Chinese and Chinese-English cross-lingual retrieval tests. Specifically designed for the RAG scenario, MiniCPM3-RAG-LoRA outperforms models like Llama3-8B and Baichuan2-13B on multiple tasks, such as open-domain question answering.

Evaluation Results Copy Chapter

Comprehensive Evaluation Copy Section

Function Calling Copy Section

We evaluate the function calling capability of MiniCPM3 on Berkeley Function Calling Leaderboard (BFCL). MiniCPM3-4B outperforms several models with 7B-9B parameters on this leaderboard, surpassing GPT-3.5-Turbo-0125.

Long Context Capability Copy Section

In the Needle in a Haystack test with a context length of 32k, the results are shown as follows:

We also propose a divide-and-conquer long-sequence processing framework LLMxMapReduce to support text with any length. MiniCPM3xMapReduce can achieve comparable performance with GPT-4 and KimiChat.

Inference Copy Chapter

Huggingface Copy Section

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
torch.manual_seed(0)

path = 'openbmb/MiniCPM3-4B'
tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map='cuda', trust_remote_code=True)

responds, history = model.chat(tokenizer, "Write an article about Artificial Intelligence.", temperature=0.7, top_p=0.7)
print(responds)

SGLang (Recommended) Copy Section

• Installation

Refer to SGLang repo to install the latest version via source code.

• Launch a server

python -m sglang.launch_server --model openbmb/MiniCPM3-4B --trust-remote-code --port 30000 --chat-template chatml

• Example code

from sglang import function, system, user, assistant, gen, set_default_backend, RuntimeEndpoint

@function
def multi_turn_question(s, question_1, question_2):
    s += user(question_1)
    s += assistant(gen("answer_1", max_tokens=1024))
    s += user(question_2)
    s += assistant(gen("answer_2", max_tokens=1024))

set_default_backend(RuntimeEndpoint("http://localhost:30000"))

state = multi_turn_question.run(
    question_1="Introduce artificial intelligence",
    question_2="Write an article about it",
)

for m in state.messages():
    print(m["role"], ":", m["content"])

vLLM Copy Section

• Install vllm
  shell Copy

  pip install "vllm>=0.6.2"

• Inference
  python Copy

  from transformers import AutoTokenizer
  from vllm import LLM, SamplingParams
  
  model_name = "openbmb/MiniCPM3-4B"
  prompt = [{"role": "user", "content": "Write an article about Artificial Intelligence."}]
  
  tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
  input_text = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True)
  
  llm = LLM(model=model_name,
      trust_remote_code=True,
      tensor_parallel_size=1
  )
  sampling_params = SamplingParams(top_p=0.7, temperature=0.7, max_tokens=1024)
  
  outputs = llm.generate(prompts=input_text, sampling_params=sampling_params)
  
  print(outputs[0].outputs[0].text)

llama.cpp Copy Section

We have provided the GGUF formats of MiniCPM3, which can be used in llama.cpp.

• Install llama.cpp
  shell Copy

  git clone https://github.com/ggerganov/llama.cpp
    cd llama.cpp
    make

• Inference
  shell Copy

  ./llama-cli -c 1024 -m minicpm3-4b-fp16.gguf -n 1024 --top-p 0.7 --temp 0.7 --prompt "<|im_start|>user\nWrite an article about Artificial Intelligence.<|im_end|>\n<|im_start|>assistant\n"

Fine-Tuning Copy Chapter

LLaMA-Factory Copy Section

We have supported fine-tuning MiniCPM3 using LLaMA-Factory. For usage instructions, refer to LLaMA-Factory Fine-tuning."

Advanced Features Copy Chapter

We use vLLM in the example code for the following advanced features.

Function calling Copy Section

We provide example code for using function calls with MiniCPM3:

cd demo/minicpm3/function_call
python function_call.py

If you want to start a function call service, use the following commands:

cd demo/minicpm3/function_call
pip install -r requirements.txt
python openai_api_server.py \
    --model openbmb/MiniCPM3-4B \
    --served-model-name MiniCPM3-4B \
    --chat-template chatml.jinja \
    --dtype auto \
    --api-key token-abc123 \
    --tensor-parallel-size 1 \
    --trust-remote-code

Below is a demo of using a search engine to answer the question:

Code Interpreter Copy Section

We provide example code for using the code interpreter with MiniCPM3:

cd demo/minicpm3/code_interpreter
pip install -r requirements.txt
python code_interpreter.py openbmb/MiniCPM3-4B

Below is an example of using the code interpreter to generate a QR code:

Deployment Arclight (`docs/deployment/arclight.md`) Copy Section

Deploy ArcLight from Source (CPU)

ArcLight is a lightweight LLM inference framework written in C/C++ for unified-memory systems. It is designed for inference scenarios beyond high-performance GPU servers, with current v1.0 optimizations focused on many-core CPU platforms and cross-NUMA tensor parallelism.

ArcLight currently supports CPU backends for ARM and x86 platforms, with basic Windows build support. The recommended path today is to build from source and run GGUF models locally.

TL;DR Copy Chapter

git clone https://github.com/OpenBMB/ArcLight.git
cd ArcLight

cmake -B build -DARCLIGHT_BACKEND=AUTO -DNNML_USE_NUMA=OFF
cmake --build build --config Release -j 32

./build/al-gen \
    --model /path/to/MiniCPM5-2B-Q4_0.gguf \
    --prompt "Hello!" \
    --numa none --nodes 1 \
    --threads 4

Preparing a Model Copy Chapter

ArcLight uses the GGUF model format from llama.cpp. The current nnml backend only ships kernels for f32 / f16 / q4_0 / q8_0 / q6_K / q8_K (see nnml/src/ops/types.cpp); other quants such as Q4_K_M will not load.

The official openbmb/MiniCPM5-2B-GGUF repo publishes F16, Q8_0, and Q4_K_M — note that Q4_0 is not included. For a q4_0 build you have to quantize it yourself from the released F16:

huggingface-cli download openbmb/MiniCPM5-2B-GGUF MiniCPM5-2B-F16.gguf --local-dir .
llama-quantize ./MiniCPM5-2B-F16.gguf ./MiniCPM5-2B-Q4_0.gguf Q4_0

The current codebase includes model definitions for:

• MiniCPM5-2B
• Qwen3
• Llama2

For first-time testing, start with MiniCPM5-2B or another small GGUF model, preferably the locally produced Q4_0 (smallest) or the released Q8_0.

Building Copy Chapter

Linux / x86 / ARM Copy Section

git clone https://github.com/OpenBMB/ArcLight.git
cd ArcLight

cmake -B build -DARCLIGHT_BACKEND=AUTO
cmake --build build --config Release -j 32

ARCLIGHT_BACKEND can be set to:

Make sure your machine has a C++17-capable toolchain, such as GCC/G++ on Linux or MSVC on Windows.

Windows Copy Section

git clone https://github.com/OpenBMB/ArcLight.git
cd ArcLight

cmake -B build -G "Visual Studio 18 2026"
cmake --build build --config Release -j 32

On Windows, executables are emitted under the build output directory configured by CMake, typically build\bin for Visual Studio builds.

Inference Copy Chapter

ArcLight currently provides three command-line apps:

• al-gen: one-shot generation
• al-chat: interactive chat
• al-ppl: perplexity evaluation for one text

If you build from source, run them from the build directory, for example ./build/al-gen on Linux.

One-shot Generate Copy Section

./build/al-gen \
    --model /path/to/MiniCPM5-2B-Q4_0.gguf \
    --prompt "Explain what unified memory means in one sentence." \
    --numa none --nodes 1 \
    --threads 4 \
    --max_length 4096 \
    --max_gen 256

For a Chinese prompt:

./build/al-gen \
    --model /path/to/MiniCPM5-2B-Q4_0.gguf \
    --prompt "用一句话解释什么是统一内存。" \
    --numa none --nodes 1 \
    --threads 4 \
    --max_gen 256

Interactive Chat Copy Section

./build/al-chat \
    --model /path/to/MiniCPM5-2B-Q4_0.gguf \
    --numa none --nodes 1 \
    --threads 4 \
    --max_length 4096 \
    --max_gen 512

Press Ctrl+C during generation to interrupt the current response. Press Ctrl+C while waiting for input to exit and print the performance profile.

Perplexity Copy Section

./build/al-ppl \
    --model /path/to/MiniCPM5-2B-Q4_0.gguf \
    --prompt "Good morning, Miss Lee!" \
    --numa none --nodes 1 \
    --threads 4

The app prints the evaluated text and a final perplexity: ... line.

NUMA Modes Copy Chapter

ArcLight supports single-node inference and cross-node tensor parallelism.

For tensor parallelism, --nodes should be a power of 2 in the current version. Choose --threads so that it can be evenly divided across NUMA nodes.

Example for a 4-node many-core machine:

./build/al-gen \
    --model /path/to/MiniCPM5-2B-Q4_0.gguf\
    --prompt "Hello!" \
    --numa tp --nodes 4 \
    --threads 32

Recommended Settings Copy Chapter

Q&A Copy Chapter

The program aborts immediately in single-node mode Copy Section

Use --numa none --nodes 1. The current implementation requires --nodes to be exactly 1 when --numa none is selected.

Tensor parallel mode fails to start Copy Section

Use --numa tp --nodes N with N > 1. In the current version, N should be a power of 2. Also make sure --threads is large enough and divisible by --nodes.

Pipeline parallelism does not work Copy Section

--numa pp is planned but not implemented yet. Use --numa none or --numa tp.

Model loading fails Copy Section

Check that the model is a GGUF checkpoint from a supported model family. The current codebase includes Qwen3, Llama, and MiniCPM5 definitions. Also verify that --w_gb is large enough for the selected model.

Inference runs out of memory Copy Section

Increase --a_gb, --kv_gb, or --work_gb. Longer contexts require a larger KV cache, so --max_length 8192 generally needs a larger --kv_gb than --max_length 4096.

See Also Copy Chapter

• ArcLight README
• llama.cpp GGUF documentation
• MiniCPM5 MLX deployment guide

Deployment Litert (`docs/deployment/litert.md`) Copy Section

Deploy MiniCPM5-2B and MiniCPM5-1B with LiteRT-LM (Android / iOS / desktop / IoT)

LiteRT-LM is Google's on-device runtime for language models, built on LiteRT (formerly TensorFlow Lite). A model is packed once into a .litertlm bundle, and the same file runs on the CPU and the GPU of Android phones, iPhones, desktops and IoT boards such as the Raspberry Pi, through a command-line tool, an OpenAI-compatible local server, a Python package, and Kotlin / Swift APIs. Use this path when the target is a phone, an app, or a local server on a laptop.

The MiniCPM5-2B bundles are hosted in litert-community/MiniCPM5-2B; that card carries the conversion recipe and the measurements quoted here. The MiniCPM5-1B bundles are hosted in litert-community/MiniCPM5-1B. Every command on this page was run with litert-lm 0.17.0 on an Apple M4 Max; the Android check used litertlm-android 0.17.0 on a Galaxy S26.

TL;DR Copy Chapter

uv tool install litert-lm

# first run downloads the 1.55 GB int4 bundle
litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-2B MiniCPM5-2B_int4.litertlm \
    --prompt "What is the capital of France?"

The file lands in ~/.litert-lm/cache/huggingface/litert-community/MiniCPM5-2B/; later runs skip the download. GPU, the thinking switches, and sampling:

MODEL=~/.litert-lm/cache/huggingface/litert-community/MiniCPM5-2B/MiniCPM5-2B_int4.litertlm

litert-lm run "$MODEL" --backend gpu --prompt "1+1=?"
litert-lm run "$MODEL" --backend gpu --thinking false --prompt "1+1=?"
litert-lm run "$MODEL" --backend gpu --thinking-budget 2048 --prompt "1+1=?"
litert-lm run "$MODEL" --top-k 40 --top-p 0.95 --temperature 1.0 --thinking false --prompt "1+1=?"

Pre-converted bundles Copy Chapter

Which 2B file: int4 is the phone file (smaller, fastest GPU decode on both devices measured) and the right one for direct answers or short reasoning. int8 is the file when the reasoning has to complete: on the same questions its thinking chains are 3–4× shorter than int4's and close where int4's keep going. Both 2B files embed the checkpoint's own chat_template.jinja, so enable_thinking and the tool-calling format work unchanged, and both declare the thought channel (next section). The 2B repo also carries two CPU-only files described in its card.

The int8 file, here on the Mac's Metal GPU:

litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-2B MiniCPM5-2B_int8.litertlm --backend gpu \
    --prompt "A train travels 60 km in 45 minutes. What is its average speed in km/h?"

MiniCPM5-1B:

litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-1B minicpm_wi4b32_wi8_afp32.litertlm --prompt "1+1=?"
litert-lm run --from-huggingface-repo=litert-community/MiniCPM5-1B minicpm_wi4b32_wi8_afp32_gpu_opt.litertlm --backend gpu --prompt "1+1=?"

Thinking Copy Chapter

Thinking is the default of the 2B files and the two 1B int4 files: with no thinking flag the model decides for itself and, in practice, reasons before every answer (MiniCPM5-1B_dynamic_wi8_afp32 is packaged with thinking off by default). The bundles declare the reasoning as a thought channel, so the CLI prints it between [thought] and [/thought] and the Kotlin / Swift APIs hand it to you separately (channels["thought"]); the answer text stays clean.

• --thinking false: direct answers with no reasoning block; on 1+1=? (GPU) the turn took 3.0 s against 24.7 s with thinking on.
• --thinking-budget N: caps the reasoning at N tokens. This model's chains run long, so start at 2048 and use 4096 for math.
• Sampling: OpenBMB recommends temperature 1.0, top_p 0.95. The bundles ship a greedy sampler (temperature 0, no top-k), which stays greedy whatever the temperature, so pass all three: --top-k 40 --top-p 0.95 --temperature 1.0 (with --top-p --temperature alone the output stays byte-identical across seeds). The card's correctness numbers are greedy.
• The KV cache takes the bundle's setting: 4096 tokens for the two 2B files and MiniCPM5-1B_dynamic_wi8_afp32, 1024 for the two 1B int4 files (--max-num-tokens to change it); the prompt format is ChatML, from the template embedded in each bundle.

OpenAI-compatible server Copy Chapter

litert-lm serve exposes every bundle in the local registry on /v1/models and /v1/chat/completions (server guide). Import the file once under a name, start the server, then call it like any OpenAI endpoint:

litert-lm import --from-huggingface-repo=litert-community/MiniCPM5-2B MiniCPM5-2B_int4.litertlm minicpm5-2b
litert-lm serve --host 127.0.0.1 --port 9379

curl http://127.0.0.1:9379/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "minicpm5-2b", "messages": [{"role": "user", "content": "1+1=?"}], "reasoning_effort": "none"}'

"reasoning_effort": "none" gives a direct answer; any other value (low to xhigh) turns thinking on, and the reasoning is counted in usage.completion_tokens_details.reasoning_tokens. With thinking on, the reasoning needs room (the int4 file filled the 4096-token context on 1+1=? and returned no content); when the reasoning has to complete, use the int8 file. max_completion_tokens caps the reply, temperature, top_p, top_k and seed are honored, and "stream": true streams.

Android Copy Chapter

AI Edge Gallery (no code). Install Google AI Edge Gallery, open Model manager, tap + and choose Import from HF, then paste the Hugging Face link of the .litertlm file (or download the file to the phone first and use From local model file). The litert-community card and the Gallery wiki have the details.

Kotlin API (your own app). The runtime is one dependency from Google Maven (Android guide):

// build.gradle.kts
implementation("com.google.ai.edge.litertlm:litertlm-android:0.17.0")

The GPU backend needs these entries in AndroidManifest.xml (Android guide); apps targeting Android 12 (API 31) or higher cannot load vendor libraries they do not declare, and without them initialize() succeeds and the first message fails with Can not find OpenCL library on this device:

<application>
    <uses-native-library android:name="libvndksupport.so" android:required="false" />
    <uses-native-library android:name="libOpenCL.so" android:required="false" />
</application>

import com.google.ai.edge.litertlm.*

val engine = Engine(EngineConfig(
    modelPath = modelFile.absolutePath,
    backend = Backend.GPU(),
    cacheDir = context.cacheDir.absolutePath,
))
engine.initialize()

engine.createConversation(ConversationConfig(maxOutputToken = 1024)).use { conversation ->
    val reply = conversation.sendMessage("Explain on-device AI in simple terms.")
    println(reply)
    println(reply.channels["thought"])
}
engine.close()

Tested on a Galaxy S26 (Snapdragon SM8850, Adreno) with litertlm-android 0.17.0 and the int4 file on Backend.GPU() (Backend.CPU() selects the CPU): the engine initializes with every node of every signature delegated to OpenCL, and the reply arrives with the reasoning in channels["thought"]. ConversationConfig also takes thinkingConfig = ThinkingConfig(enableThinking = false) or ThinkingConfig(enableThinking = true, thinkingTokenBudget = 2048) and a samplerConfig = SamplerConfig(topK = 40, topP = 0.95, temperature = 1.0), as in the Android guide. conversation.sendMessageAsync(...) streams.

iOS Copy Chapter

Tested on an iPhone 17 Pro with the int4 file on both backends (Metal GPU and CPU; init 5.7 s and 2.2 s). The int8 file's main weight section is 2.33 GB, more than an iOS app maps in one piece with the default entitlements; the com.apple.developer.kernel.increased-memory-limit entitlement lifts that limit (single sections of 2.87 GB and 4.24 GB from other models have loaded and run on an iPhone 17 Pro with it), not checked with this file. The int4 file's main section is 1.28 GB. The Swift API is documented in the LiteRT-LM Swift guide (Swift Package Manager, https://github.com/google-ai-edge/LiteRT-LM); this page does not cover the Xcode steps.

Speed and accuracy Copy Chapter

All numbers are from the litert-community card. Apple M4 Max, litert-lm benchmark 0.17.0, 256-token prefill / 256-token decode, 3 runs, 1024-token KV cache, no compiled-model cache (--cache no):

Galaxy S26 (Snapdragon SM8850, Adreno), the LiteRT-LM v0.16.0 release binary, 205-token prompt, 2 runs per cell (ranges shown; a reasoning model decodes its own full response, so decode lengths vary):

Accuracy, GSM8K first 100 test questions, greedy, thinking off, up to 2048 new tokens: bf16 PyTorch 92 %, int8 CPU 91 %, int4 CPU 86 %, int4 GPU 87 %. Both 2B files score 8/8 on the card's 8-question sanity gate on the Mac, CPU and GPU.

Common pitfalls Copy Chapter

• Thinking on the int4 file runs long chains: on the card's ten-question thinking-on subset (3584-token budget) it closes 0/10 on the CPU, where int8 closes 9/10 like the bf16 model. Pass --thinking false for direct answers; when the reasoning has to complete, use the int8 file.
• int8 on iOS needs the increased-memory-limit entitlement. With the default entitlements the 2.33 GB weight section does not map; add com.apple.developer.kernel.increased-memory-limit (see iOS). int4's 1.28 GB main section is under the limit.
• The output cap is what leaves no answer: when the reasoning runs into the context (--max-num-tokens, 4096 for the 2B files) or the app's maxOutputToken, the chain is cut and nothing follows. When the reasoning has to complete, keep those at 2048 or more (4096 for math); 1024, as in the snippets, is enough for short answers.
• First run per backend is slow: the CPU run writes an XNNPACK weight cache beside the file and the GPU run compiles its kernels (--cache no skips the disk cache; the Kotlin cacheDir is the same mechanism).
• GPU in your own Android app: without the `` entries above, initialize() succeeds and the first message fails with Can not find OpenCL library on this device.
• Temperature without top-k does nothing: --temperature and --top-p only take effect together with --top-k above 1 (the default is greedy). The Kotlin SamplerConfig takes the same three fields.
• Scripts: the CLI reads stdin; when it runs from a script or a job with an open pipe on stdin, redirect it (< /dev/null) or it waits for end-of-input before generating.

See also Copy Chapter

• litert-community/MiniCPM5-2B — recipe, correctness, all measurements, litertlm_manifest.json
• litert-community/MiniCPM5-1B
• LiteRT-LM guides: CLI, OpenAI-compatible server, CLI configuration, model management, Android, Swift, Python

Deployment Llama Cpp (`docs/deployment/llama_cpp.md`) Copy Section

Deploy MiniCPM5-1B and MiniCPM5-2B with llama.cpp

llama.cpp is the recommended path for CPU / edge / consumer-GPU deployment. The released GGUF builds run on laptops, single-board computers, Apple Silicon, and Windows boxes with no Python at all.

Released GGUF artifacts Copy Chapter

These artifacts work directly with vanilla llama.cpp and every llama.cpp-based runtime (Ollama / LM Studio / llama-cpp-python).

TL;DR — run a release GGUF Copy Chapter

huggingface-cli download openbmb/MiniCPM5-2B-GGUF MiniCPM5-2B-Q4_K_M.gguf --local-dir ./minicpm5

# Interactive chat (auto-applies the chat template)
llama-cli -m ./minicpm5/MiniCPM5-2B-Q4_K_M.gguf -n 2048 --temp 1.0 --top-p 0.95 --min-p 0.0 -ngl 99

OpenAI-compatible server Copy Chapter

llama-server -m MiniCPM5-2B-Q4_K_M.gguf --port 8080 -ngl 99 -c 8192 --jinja

curl http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "MiniCPM5-2B",
        "messages": [{"role": "user", "content": "1+1=?"}],
        "temperature": 1.0, "top_p": 0.95, "min_p": 0.0, "max_tokens": 256
    }'

In llama.cpp, the default min_p=0.05 can lead to repetitive output: it filters out tokens whose probability is below 5% of the highest-probability token, potentially discarding the exact tokens needed to break out of a repetition loop. To prevent this, we set min_p=0.0.

Generation parameters Copy Chapter

Build a GGUF from your own checkpoint Copy Chapter

If you've trained your own MiniCPM5-1B or MiniCPM5-2B variant (continue-pretraining, domain SFT, …) and want to publish a GGUF, the pipeline is:

git clone --depth=1 https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
mkdir -p build && cd build

# CPU-only build (sufficient for quantize + sanity check)
cmake .. -DGGML_CUDA=OFF -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build . --config Release -j $(nproc) --target llama-quantize llama-cli llama-server

# Or a CUDA build for high-throughput inference
# cmake .. -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=90 -DCMAKE_BUILD_TYPE=Release
# (set CMAKE_CUDA_ARCHITECTURES to your GPU compute capability, see NVIDIA docs)

cd ..
SRC=/path/to/your-MiniCPM5-fp16-hf
OUT=/path/to/output

# Run from the llama.cpp repository root cloned above.
python ./convert_hf_to_gguf.py "$SRC" --outfile "$OUT/F16.gguf" --outtype f16
build/bin/llama-quantize "$OUT/F16.gguf" "$OUT/Q4_K_M.gguf" Q4_K_M
build/bin/llama-quantize "$OUT/F16.gguf" "$OUT/Q8_0.gguf"   Q8_0

See also Copy Chapter

• ollama.md — ollama run directly from these GGUFs
• lmstudio.md — desktop GUI for the same GGUFs

Deployment Lmstudio (`docs/deployment/lmstudio.md`) Copy Section

Deploy MiniCPM5-1B and MiniCPM5-2B with LM Studio

LM Studio is the GUI-first path for running MiniCPM5-1B or MiniCPM5-2B on a Mac / Windows / Linux laptop — drag a model in, click Load, chat, or expose an OpenAI-compatible REST endpoint with one toggle. On Apple Silicon it ships two inference runtimes that work for both models:

LM Studio does not load raw Hugging Face transformers checkpoints directly. To use the MLX runtime, point LM Studio at a pre-converted MLX repo, or run mlx_lm.convert first — see mlx.md for the full pipeline.

TL;DR Copy Chapter

# 1. Install LM Studio (one-time) and run it once to complete onboarding.
brew install --cask lm-studio
open -a "LM Studio"          # accept EULA + pick model source

# 2. Drop the released GGUF into LM Studio's model registry.
#    The expected layout is <publisher>/<repo>/<file>.gguf:
mkdir -p ~/.lmstudio/models/openbmb/MiniCPM5-2B-GGUF
cp MiniCPM5-2B-Q4_K_M.gguf ~/.lmstudio/models/openbmb/MiniCPM5-2B-GGUF/

# 3. Use the bundled `lms` CLI to start the local OpenAI-compatible server.
LMS="/Applications/LM Studio.app/Contents/Resources/app/.webpack/lms"
"$LMS" ls
"$LMS" server start                                   # binds 127.0.0.1:1234
"$LMS" load minicpm5-2b --gpu max --context-length 8192 -y
"$LMS" ps

💡 The very first launch needs the GUI — lms will refuse with Cannot find LM Studio installation until LM Studio has run interactively at least once. Subsequent sessions are fully scriptable.

Importing the model Copy Chapter

LM Studio scans ~/.lmstudio/models/ recursively for *.gguf, but only displays models whose path matches /<model>/<file>.gguf. Two equivalent options:

# Option A — drop into the registry directly (recommended for scripted envs)
mkdir -p ~/.lmstudio/models/openbmb/MiniCPM5-2B-GGUF
cp MiniCPM5-2B-Q4_K_M.gguf ~/.lmstudio/models/openbmb/MiniCPM5-2B-GGUF/

# Option B — drag-and-drop into the LM Studio GUI's "My Models" view.

After either path, lms ls should show:

LLM              PARAMS    ARCH     SIZE       DEVICE
minicpm5-2b       2B        Llama    1.56 GB    Local

Loading and serving Copy Chapter

LMS="/Applications/LM Studio.app/Contents/Resources/app/.webpack/lms"

"$LMS" server start
"$LMS" load minicpm5-2b \
    --gpu max \
    --context-length 8192 \
    -y

--gpu max offloads every transformer layer to Metal; the M4's unified memory model means you don't need to budget VRAM separately. Reduce to --gpu 0.5 or similar on machines where you want to leave more memory for other apps.

Verify with:

"$LMS" ps
# IDENTIFIER       MODEL          STATUS    SIZE       CONTEXT    PARALLEL    DEVICE    TTL
# minicpm5-2b      minicpm5-2b    IDLE      1.56 GB    8192       4           Local

Inference Copy Chapter

LM Studio exposes an OpenAI-compatible API on http://localhost:1234/v1. The chat template baked into the GGUF is auto-applied:

Short prompt Copy Section

curl -sS http://localhost:1234/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "minicpm5-2b",
        "messages": [{"role": "user", "content": "用一句话解释什么是 GQA。"}],
        "temperature": 1.0, "top_p": 0.95, "max_tokens": 200,
        "stop": ["<|im_end|>", "<|im_start|>"]
    }'

Think (reasoning) Copy Section

curl -sS http://localhost:1234/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "minicpm5-2b",
        "messages": [{"role": "user", "content": "鸡兔同笼，头共10个，脚共28只，问鸡和兔各几只？请逐步推理后给出答案。"}],
        "temperature": 1.0, "top_p": 0.95, "max_tokens": 400,
        "stop": ["<|im_end|>", "<|im_start|>"]
    }'

💡 On Apple Silicon the MLX Q4 build is the clear winner: ~60% faster than the equivalent GGUF Q4_K_M, reasoning-tokens are routed to OpenAI's reasoning_content extension (so the visible content only contains the final answer), and the math answer matches F16 quality. Use the GGUF build only when cross-platform parity matters.

Auto think/answer split (MLX runtime only) Copy Section

When LM Studio's mlx-llm runtime detects a ... block in the model's output, it splits the OpenAI response automatically:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": "鸡有6只，兔子有4只 …",
      "reasoning_content": "我们被问到：…设鸡的数量为x …"
    },
    "finish_reason": "stop"
  }],
  "usage": {
    "completion_tokens": 1428,
    "completion_tokens_details": {"reasoning_tokens": 1147}
  }
}

The GGUF runtime does not do this split — ... is emitted inline as part of content, and you have to strip it client-side. If your downstream client supports the OpenAI reasoning extension (Cursor, Continue, etc.), prefer the MLX runtime.

💡 If you observe the model continuing past <|im_end|> into a hallucinated second turn, add "stop": ["<|im_end|>", "<|im_start|>"] to the request. <|im_end|> is token id 130073 and is already in the GGUF's eos_token_id array, so LM Studio normally stops there on its own — only resort to an explicit stop array if you actually see overrun in practice.

Recommended sampling Copy Chapter

Toggling Think / No-think (MiniCPM5-1B) Copy Chapter

As of 2026-05, LM Studio's OpenAI-compatible layer does not consistently honour the standard chat_template_kwargs.enable_thinking flag. Passing it in the request body may have no effect, and natural-language hints (/no_think suffix, system prompt saying "do not include reasoning", etc.) do not reliably suppress thinking either. So in the GUI chat pane there may be no built-in toggle.

The cleanest workaround is to register two extra model variants with the enable_thinking switch hard-coded into the Jinja template, then switch modes by switching model in the GUI's model dropdown. Variant folders symlink the weights, so each variant adds ~10 KB on disk (one rewritten chat_template.jinja).

One-time setup script Copy Section

The script below operates on already-imported MLX folders under ~/.lmstudio/models/openbmb/. Get them there first — there is no pre-converted MiniCPM5-1B-MLX-Q4 / MiniCPM5-1B-MLX-bf16 Hugging Face repo to clone; the only published MLX repo is openbmb/MiniCPM5-1B-MLX (4-bit affine). Either drop it in as-is, or build the two flavours locally with mlx_lm.convert (see mlx.md) and mv them into the LM Studio registry, e.g.:

# Option A — use the official 4-bit affine repo as a single source
mkdir -p ~/.lmstudio/models/openbmb/MiniCPM5-1B-MLX
huggingface-cli download openbmb/MiniCPM5-1B-MLX \
    --local-dir ~/.lmstudio/models/openbmb/MiniCPM5-1B-MLX

# Option B — build both flavours from the fp16 HF repo
HF=openbmb/MiniCPM5-1B
mlx_lm.convert --hf-path "$HF" --mlx-path ~/.lmstudio/models/openbmb/MiniCPM5-1B-MLX-bf16
mlx_lm.convert --hf-path "$HF" --mlx-path ~/.lmstudio/models/openbmb/MiniCPM5-1B-MLX-Q4 -q --q-bits 4

Then list the flavours you want the variant script to touch in SOURCES.

python3 - <<'PY'
import os, re, shutil

ROOT = os.path.expanduser("~/.lmstudio/models/openbmb")
# Set to whichever flavour folders actually exist under ROOT:
SOURCES = ["MiniCPM5-1B-MLX"]  # or ["MiniCPM5-1B-MLX-Q4", "MiniCPM5-1B-MLX-bf16"]

block_re = re.compile(
    r"\{%-\s*if\s+enable_thinking\s+is\s+defined\s*%\}.*?"
    r"\{%-\s*endif\s*%\}\s*\{%-\s*endif\s*%\}",
    re.DOTALL,
)

def make_variant(src, dst, mode):
    if os.path.exists(dst): shutil.rmtree(dst)
    os.makedirs(dst, exist_ok=True)
    for f in os.listdir(src):
        sp, dp = os.path.join(src, f), os.path.join(dst, f)
        if f == "chat_template.jinja":
            tpl = open(sp).read()
            replacement = (
                "{{- '<think>\\n\\n</think>\\n\\n' }}"  # nothink
                if mode == "nothink"
                else "{{- '<think>\\n' }}"             # think
            )
            new_tpl, n = block_re.subn(replacement, tpl, count=1)
            assert n == 1, "enable_thinking block not found in template"
            open(dp, "w").write(new_tpl)
        else:
            os.symlink(os.path.realpath(sp), dp)

for src_name in SOURCES:
    src = f"{ROOT}/{src_name}"
    make_variant(src, f"{ROOT}/{src_name}-NoThink", "nothink")
    make_variant(src, f"{ROOT}/{src_name}-Think",   "think")
print("done — restart LM Studio (or run `lms ls`) to re-scan the registry.")
PY

After running it, lms ls shows four extra rows alongside the originals:

LLM                               PARAMS    ARCH     SIZE         DEVICE
minicpm5-1b-mlx@4bit            1B      Llama    617.97 MB    Local      ← jinja default (think)
minicpm5-1b-mlx-nothink@4bit    1B      Llama    617.97 MB    Local      ← always no-think
minicpm5-1b-mlx-think@4bit      1B      Llama    617.97 MB    Local      ← always think
…and bf16 counterparts

In the GUI chat pane just pick the variant you want from the model dropdown — the chat template baked into each variant decides whether `` activates.

Variant behaviour example (`用一句话告诉我中国首都。`, `max_tokens=120`) Copy Section

⚠️ Don't try to write the assistant pre-fill trick ({"role":"assistant","content":"\n\n\n\n"}) from the GUI — LM Studio's chat pane doesn't allow appending an assistant turn before send. The variant approach is the only path that works in pure-GUI use. From the OpenAI HTTP API, the assistant pre-fill also works and avoids needing extra variants.

Q&A Copy Chapter

`lms` says `Cannot find LM Studio installation` Copy Section

You haven't completed LM Studio's first-run GUI onboarding yet. Open the app from Launchpad, accept the EULA + pick a model source, then re-run the CLI command.

`lms import` hangs with no output Copy Section

Skip lms import and use the registry-drop path (Option A above) — cp the GGUF into ~/.lmstudio/models//<repo>/. LM Studio picks it up within a few seconds without any further command.

Output looks like the assistant impersonates the user / continues past one turn Copy Section

You forgot the explicit stop array — see the sanity-check snippets above.

See also Copy Chapter

• ollama.md — the same GGUF, CLI / daemon path
• llama_cpp.md — the underlying engine; build / quantize recipe
• mlx.md — alternative MLX path on Apple Silicon (no GUI)

Deployment Mlx (`docs/deployment/mlx.md`) Copy Section

Deploy MiniCPM5-1B and MiniCPM5-2B with MLX (Apple Silicon)

MLX is Apple's on-device tensor framework. For MiniCPM5-1B and MiniCPM5-2B it is the recommended path on Apple Silicon (M1–M4) when you want highest throughput and want to stay inside one Python process — no separate server, no llama.cpp build chain.

TL;DR Copy Chapter

pip install "mlx-lm>=0.31"

# Run the official pre-converted MLX repo directly
# (config.json declares "quantization": {"bits": 4, "mode": "affine"})
mlx_lm.generate --model openbmb/MiniCPM5-2B-MLX \
    --prompt "<|im_start|>user
1+1=?<|im_end|>
<|im_start|>assistant
" \
    --max-tokens 200 --temp 1.0 --top-p 0.95

Building MLX weights from your own checkpoint (advanced) Copy Chapter

If you have a self-trained HF fp16 checkpoint and want to produce MLX weights, use mlx_lm.convert:

HF=/path/to/your-fp16-hf

# bf16 master copy
mlx_lm.convert --hf-path "$HF" --mlx-path ./minicpm5-mlx-bf16

# 4-bit (smaller / faster, ~4.5 bits/weight on average)
mlx_lm.convert --hf-path "$HF" --mlx-path ./minicpm5-mlx-q4 -q --q-bits 4

The 4-bit pass logs [INFO] Quantized model with 4.501 bits per weight. — the slight overshoot above 4 bits is from keeping embed_tokens and lm_head in higher precision, which preserves quality on the small, untied vocabulary head.

Inference Copy Chapter

One-shot generate Copy Section

mlx_lm.generate --model ./minicpm5-mlx-q4 \
    --prompt "<|im_start|>user
鸡兔同笼，头共10个，脚共28只，问鸡和兔各几只？<|im_end|>
<|im_start|>assistant
" \
    --max-tokens 200 --temp 1.0 --top-p 0.95

首先，理解问题：总共有10个头（鸡和兔都是头），总共有28只脚。我们需要找出鸡和兔各自的数量。

设鸡的数量为x，兔的数量为y。那么：
1. 头数：总头数 = 鸡的数量 + 兔的数量 = x + y = 10。
2. 脚数：鸡有2只脚，兔有4只脚，总脚数 = 2x + 4y = 28。
…

Python API (streaming) Copy Section

from mlx_lm import load, stream_generate

model, tk = load("./minicpm5-mlx-q4")

prompt = (
    "<|im_start|>user\n"
    "用一句话解释什么是 GQA。<|im_end|>\n"
    "<|im_start|>assistant\n"
)

for resp in stream_generate(
    model, tk, prompt=prompt, max_tokens=512,
    sampler=None,                    # default temp/top_p
):
    print(resp.text, end="", flush=True)
print()

ℹ️ <|im_end|> is token id 130073 in this tokenizer, and generation_config.json already lists it in eos_token_id: [1, 130073]. mlx-lm reads that list and adds both ids to the stop set, so no --extra-eos-token flag is needed — the model stops at end-of-turn on its own.

Recommended sampling Copy Chapter

For MiniCPM5-1B, both modes are activated by sampling parameters only — the released chat template auto-injects \n when no system message disables it, so you get think-mode behaviour by default.

Q&A Copy Chapter

Model never stops generating Copy Section

Confirm you're on mlx-lm >= 0.31. Older versions ignored multi-id eos_token_id lists in generation_config.json and would only stop at </s> (id 1) — on a chat-template turn the model never emits </s> so it ran past the turn boundary. 0.31+ honours the full list ([1, 130073]) and stops at <|im_end|> automatically. As a last-resort override you can still pass --extra-eos-token "<|im_end|>".

See also Copy Chapter

• transformers.md — same checkpoint, CPU / CUDA path
• llama_cpp.md — alternative on-device path (CPU + Metal)
• lmstudio.md — desktop GUI consumer of the same MLX models

Deployment Ollama (`docs/deployment/ollama.md`) Copy Section

Deploy MiniCPM5-1B and MiniCPM5-2B with Ollama

Ollama is the easiest CLI / daemon path to run MiniCPM5-1B or MiniCPM5-2B on a laptop or desktop — one binary, no Python, no CUDA toolkit. It consumes the same GGUF files we ship for llama.cpp.

TL;DR Copy Chapter

brew install ollama                 # macOS
# or: curl -fsSL https://ollama.com/install.sh | sh   (Linux)

OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve &

# Pull the released GGUF and create the Ollama model
mkdir -p ~/minicpm5-2b && cd ~/minicpm5-2b
huggingface-cli download openbmb/MiniCPM5-2B-GGUF MiniCPM5-2B-Q4_K_M.gguf --local-dir .

QUANT=Q4_K_M
cat > Modelfile <<EOF
FROM ./MiniCPM5-2B-${QUANT}.gguf

# MiniCPM5 basic chat template
TEMPLATE """{{- if .Messages -}}
{{- range .Messages -}}
<|im_start|>{{ .Role }}
{{ .Content }}<|im_end|>
{{ end -}}
<|im_start|>assistant
{{ end -}}"""

# Stop on either EOS or the chat-template terminator
PARAMETER stop "<|im_end|>"
PARAMETER stop "</s>"

# Defaults are tuned for think mode
PARAMETER temperature 1.0
PARAMETER top_p 0.95
PARAMETER num_ctx 8192
EOF

ollama create minicpm5-2b -f Modelfile
ollama run minicpm5-2b

Choosing a quant Copy Chapter

API access Copy Chapter

Ollama serves an OpenAI-compatible REST endpoint on http://localhost:11434/v1:

curl http://localhost:11434/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "minicpm5-2b",
        "messages": [{"role": "user", "content": "用一句话解释 GQA。"}],
        "temperature": 1.0, "top_p": 0.95, "max_tokens": 1024
    }'

Or use the Ollama-native API:

curl http://localhost:11434/api/chat -d '{
    "model": "minicpm5-2b",
    "messages": [{"role":"user","content":"1+1=?"}],
    "stream": false,
    "options": {"temperature": 1.0, "top_p": 0.95}
}'

MiniCPM5-1B Think vs No-think Copy Chapter

A MiniCPM5-1B Modelfile configured with temperature=0.7, top_p=0.95 defaults to no-think. To switch a single conversation to think mode, override the sampling params at request time:

ollama run minicpm5-1b --temperature 0.9 --top-p 0.95

Or bake it into a separate model tag by raising the temperature line (top_p stays 0.95):

PARAMETER temperature 0.9
PARAMETER top_p 0.95

Then ollama create minicpm5-1b-think -f Modelfile.think.

ℹ️ Ollama 0.24 does not directly evaluate the GGUF-embedded Jinja chat template. It maps recognized Jinja templates to built-in Go templates, so MiniCPM5 uses the explicit Go TEMPLATE block above. To enter the MiniCPM5-2B think path, use raw mode and prepend \n manually:

bash Copy

curl http://localhost:11434/api/generate -d '{
    "model": "minicpm5-2b",
    "raw": true,
    "prompt": "<|im_start|>user\n鸡兔同笼…<|im_end|>\n<|im_start|>assistant\n<think>\n",
    "options": {"temperature": 1.0, "top_p": 0.95}
}'

Higher throughput on Apple Silicon Copy Chapter

For long-context workloads, raise num_ctx:

PARAMETER num_ctx 32768

For sustained throughput with larger KV caches, set OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 in the environment before ollama serve.

See also Copy Chapter

• llama_cpp.md — the engine behind Ollama; full GGUF build pipeline
• lmstudio.md — desktop GUI consumer of the same GGUFs
• mlx.md — alternative on-device path on Apple Silicon (faster for Q4)

Deployment Sglang (`docs/deployment/sglang.md`) Copy Section

Deploy MiniCPM5-1B and MiniCPM5-2B with SGLang

SGLang serves MiniCPM5-1B and MiniCPM5-2B as standard LlamaForCausalLM models with RadixAttention prefix cache, very high concurrency, and an OpenAI-compatible API.

Install Copy Chapter

pip install "sglang[srt]>=0.5.16"          # latest, requires CUDA 13.x driver
# pip install "sglang==0.5.6.post3"        # fallback for CUDA 12.x drivers

⚠️ Tool calling needs a newer build than any current pip release. The MiniCPM5 XML parser (--tool-call-parser minicpm5) landed in sgl-project/sglang#25600, merged into main on 2026-05-22. The latest release v0.5.12.post1 was branched earlier and does not ship the parser yet. Until v0.5.13 is tagged, install from source if you need tool calling:

bash Copy

pip install "git+https://github.com/sgl-project/sglang.git@main#subdirectory=python"

Plain chat completions (no tools=) work fine on the pip release — both checkpoints are stock LlamaForCausalLM models, so they load on every SGLang version that supports Llama.

Recommended runtime env vars:

export VLLM_WORKER_MULTIPROC_METHOD=spawn
export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1
export SGLANG_DISABLE_CUDNN_CHECK=1

OpenAI-compatible server Copy Chapter

python -m sglang.launch_server \
    --model-path openbmb/MiniCPM5-2B \
    --served-model-name MiniCPM5-2B \
    --dtype bfloat16 \
    --context-length 131072 \
    --mem-fraction-static 0.85 \
    --tool-call-parser minicpm5 \
    --host 0.0.0.0 \
    --port 30000

Speculative decoding (DSpark): we also release MiniCPM5-2B-DSpark, a DSpark draft model trained for MiniCPM5-2B. Enable it in SGLang to accelerate decoding while keeping the target model's outputs unchanged:

python -m sglang.launch_server \
    --model-path "${MODEL_PATH}" \
    --served-model-name MiniCPM5-2B \
    --dtype bfloat16 \
    --context-length ${CTX_LEN} \
    --mem-fraction-static ${MEM_FRAC} \
    --tool-call-parser ${TOOL_PARSER} \
    --host 0.0.0.0 \
    --port 3000 \
    --speculative-algorithm DSPARK \
    --speculative-draft-model-path openbmb/MiniCPM5-2B-DSpark \
    --speculative-dspark-block-size 7

Tuning knobs Copy Section

Chat completion Copy Chapter

curl http://localhost:30000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "MiniCPM5-2B",
        "messages": [{"role": "user", "content": "用一句话解释什么是 GQA。"}],
        "temperature": 1.0, "top_p": 0.95, "max_tokens": 1024,
        "chat_template_kwargs": {"enable_thinking": true}
    }'

Tool calling Copy Chapter

MiniCPM5-1B and MiniCPM5-2B emit XML-style tool calls. SGLang ships a built-in minicpm5 parser that converts them to OpenAI-compatible tool_calls.

Start the server with either --tool-call-parser minicpm5 or --tool-call-parser auto, then send a standard OpenAI-style tools request:

curl http://localhost:30000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "MiniCPM5-2B",
        "messages": [{"role": "user", "content": "What is the weather in Beijing?"}],
        "tools": [{
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Get current weather for a city",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"]
                }
            }
        }],
        "tool_choice": "auto",
        "temperature": 1.0, "top_p": 0.95,
        "max_tokens": 256,
        "chat_template_kwargs": {"enable_thinking": true}
    }'

Offline / batched (Engine API) Copy Chapter

import sglang as sgl

llm = sgl.Engine(
    model_path="openbmb/MiniCPM5-2B",
    tp_size=1,
    mem_fraction_static=0.8,
    context_length=131072,
)

outputs = llm.generate(
    ["用一句话解释什么是 GQA。"],
    sampling_params={
        "temperature": 1.0, "top_p": 0.95, "max_new_tokens": 1024,
        "skip_special_tokens": False,
    },
)
print(outputs)

Deployment Transformers (`docs/deployment/transformers.md`) Copy Section

Deploy MiniCPM5-1B and MiniCPM5-2B with 🤗 Transformers

MiniCPM5-1B and MiniCPM5-2B are standard LlamaForCausalLM models, so they load directly via AutoModelForCausalLM — no custom modeling code, no trust_remote_code.

Install Copy Chapter

pip install -U "transformers>=5.6,<6" "torch>=2.11" accelerate     # latest (CUDA 13.x driver hosts)
# pip install -U "transformers==4.57.3" "torch==2.7.1" accelerate  # fallback for CUDA 12.x driver hosts

GPU inference (bfloat16) Copy Chapter

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "openbmb/MiniCPM5-2B"  # or openbmb/MiniCPM5-1B
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    device_map="auto",
).eval()

messages = [{"role": "user", "content": "用一句话解释什么是 GQA。"}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    enable_thinking=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=1024,
        do_sample=True,
        temperature=1.0,
        top_p=0.95,
    )

prompt_len = inputs["input_ids"].shape[-1]
print(tokenizer.decode(outputs[0][prompt_len:], skip_special_tokens=True))

CPU-only inference Copy Chapter

Both checkpoints can run on CPU only with sufficient system RAM (for example, on laptops, CI machines, or no-GPU sanity-check hosts):

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "openbmb/MiniCPM5-2B"  # or openbmb/MiniCPM5-1B
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.float32,   # bf16 also works on AVX-512 BF16 / AMX hosts
    device_map="cpu",
).eval()

messages = [{"role": "user", "content": "用一句话解释什么是 GQA。"}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    enable_thinking=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=1024,
        do_sample=True,
        temperature=1.0,
        top_p=0.95,
    )

prompt_len = inputs["input_ids"].shape[-1]
print(tokenizer.decode(outputs[0][prompt_len:], skip_special_tokens=True))

Generation defaults Copy Chapter

generation_config.json is tuned for think mode by default.

LoRA inference (PEFT) Copy Chapter

from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained(
    "openbmb/MiniCPM5-2B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "<your_lora_dir>").eval()

Adapters trained against this base load directly with no surgery.

Deployment Vllm (`docs/deployment/vllm.md`) Copy Section

Deploy MiniCPM5-1B and MiniCPM5-2B with vLLM

vLLM >=0.21 supports MiniCPM5-1B and MiniCPM5-2B natively — no custom kernels. For production-grade throughput and OpenAI-compatible chat completions, this is the recommended path.

Install Copy Chapter

pip install "vllm>=0.21"          # latest (CUDA 13.x driver hosts)
# pip install "vllm==0.10.1.1"    # fallback for CUDA 12.x driver hosts

OpenAI-compatible server Copy Chapter

vllm serve openbmb/MiniCPM5-2B \
    --served-model-name MiniCPM5-2B \
    --dtype bfloat16 \
    --max-model-len 131072 \
    --gpu-memory-utilization 0.85 \
    --port 8000

Tuning knobs Copy Section

Chat completions Copy Chapter

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "MiniCPM5-2B",
        "messages": [{"role": "user", "content": "用一句话解释什么是 GQA。"}],
        "temperature": 1.0,
        "top_p": 0.95,
        "max_tokens": 1024,
        "chat_template_kwargs": {"enable_thinking": true}
    }'

Sample run Copy Chapter

$ curl -sS http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"MiniCPM5-2B","messages":[{"role":"user","content":"1+1=?"}],"temperature":1.0,"top_p":0.95,"max_tokens":64,"chat_template_kwargs":{"enable_thinking":true}}'
{
  "id": "chatcmpl-...",
  "model": "MiniCPM5-2B",
  "choices": [{"message": {"role": "assistant", "content": "2"}, "finish_reason": "stop"}],
  "usage": {"prompt_tokens": 14, "completion_tokens": 2, "total_tokens": 16}
}

Offline / batched inference Copy Chapter

from vllm import LLM, SamplingParams

llm = LLM(model="openbmb/MiniCPM5-2B", dtype="bfloat16", max_model_len=131072)
out = llm.chat(
    [[{"role": "user", "content": "用一句话解释 GQA。"}]],
    SamplingParams(temperature=1.0, top_p=0.95, max_tokens=512),
    chat_template_kwargs={"enable_thinking": True},
)
print(out[0].outputs[0].text)

Tool calling (plugin) Copy Chapter

MiniCPM5-1B and MiniCPM5-2B emit XML-style tool calls. The vLLM-side parser (vllm-project/vllm#43175) was merged to main on 2026-05-27 but is not in any pip release yet — v0.22.0 (2026-05-29) was cut before that merge and the file is absent from the v0.22.0 tree.

As a bridge, this repo ships the parser at tool_parsers/minicpm5xml_tool_parser.py (the same file as the upstream PR). Load it via vLLM's --tool-parser-plugin:

vllm serve openbmb/MiniCPM5-2B \
    --served-model-name MiniCPM5-2B \
    --dtype bfloat16 --max-model-len 131072 --port 8000 \
    --enable-auto-tool-choice \
    --tool-parser-plugin /path/to/MiniCPM/tool_parsers/minicpm5xml_tool_parser.py \
    --tool-call-parser minicpm5

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "MiniCPM5-2B",
        "messages": [{"role": "user", "content": "What is the weather in Beijing?"}],
        "tools": [{
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Get current weather for a city",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"]
                }
            }
        }],
        "tool_choice": "auto",
        "temperature": 1.0, "top_p": 0.95, "max_tokens": 256,
        "chat_template_kwargs": {"enable_thinking": true}
    }'

Once vLLM v0.23 (or later) is released with the parser baked in, drop --tool-parser-plugin and use only --tool-call-parser minicpm5.