MiniCPM-V Documentation

Technical Documentation: OpenBMB/MiniCPM-V

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (README.md) Copy Section

MiniCPM-V 4.7 Copy Chapter

MiniCPM-V 4.7 is the latest model in the MiniCPM-V series, available in 35B-A3B and 1B variants. The 35B-A3B variant is the series' first MoE model. Its main features include:

• 🚀 A new performance–efficiency Pareto frontier. The 35B-A3B and 1B variants each advance the performance–efficiency Pareto frontier for multimodal large language models. In Instruct mode, 35B-A3B achieves a higher overall score in less inference time than Qwen3.6 35B-A3B; the 1B variant further improves on MiniCPM-V 4.6's core capabilities and substantially outperforms Qwen3.5 0.8B in both overall score and inference efficiency.
• 📱 The first open-source MoE to run smoothly on-device without compromising performance. Powered by the latest Lookahead Expert Scheduling (LES) inference acceleration technique, MiniCPM-V 4.7 35B-A3B becomes the first open-source MoE model to run smoothly on-device without compromising performance, achieving a decoding speed of 13+ tokens/s with just 3 GB of memory on iPhone 17 Pro Max. We also support DSpark speculative decoding, with dedicated heads available for 1B (DSpark-BF16, DSpark-GGUF) and 35B-A3B (DSpark-BF16, DSpark-GGUF).
• 🛠️ Broad platform support and easy customization. MiniCPM-V 4.7 supports iOS, Android, and HarmonyOS, and supports chip platforms from Intel, AMD, Arm, Rockchip, and Houmo, covering a diverse range of mobile devices and local computing scenarios. We also open-source Self-Replay Fine-Tuning (SRFT), helping developers extend model capabilities at low cost while reducing forgetting of existing capabilities.

News Copy Section

• 2026-10-15 · We release DSpark speculative decoding heads in BF16 and GGUF for both MiniCPM-V 4.7 variants: 1B (DSpark-BF16, DSpark-GGUF) and 35B-A3B (DSpark-BF16, DSpark-GGUF).
• 2026-10-12 · MiniCPM-V 4.7 released — the first open-source MoE model to run smoothly on-device without compromising performance, setting a new multimodal performance–efficiency Pareto frontier. See what's new.

Significantly Advancing the Performance–Efficiency Pareto Frontier Copy Section

Overall Performance (35B-A3B, Instruct)

Vision-Language

Language

Overall Performance (1B, Instruct)

Smooth On-Device MoE Inference without Compromising Performance Copy Section

Running an MoE model on a device is only the starting point. The real challenge is to preserve model capability while delivering smooth generation. Storing expert weights on SSD can overcome memory capacity limits, but frequent loading stalls slow generation unless loading and computation overlap effectively. Restricting the number of experts or fixing an expert subset can speed up inference at the cost of model capability.

To address this trade-off, MiniCPM-V 4.7 introduces Lookahead Expert Scheduling, delivering a 5× speedup on real devices without sacrificing model capability. Cross-layer routing prediction preloads experts likely to be needed by the next layer and prioritizes computation for experts selected by the actual router that are already loaded, substantially reducing loading stalls. Weight quantization and an optimized storage layout also reduce the storage footprint and make scattered reads more efficient, further accelerating loading. The recording below compares generation speed before and after inference optimization:

Model Weights Copy Section

MiniCPM-V 4.7 is available in 35B-A3B and 1B variants. Each row lists the model weights alongside its corresponding DSpark speculative decoding heads, available in BF16 and GGUF.

The 1B int4 variant uses bitsandbytes; AWQ and GPTQ use W4A16 quantization. Quantized variants reduce memory usage; GGUF weights support CPU inference.

Also available on Ollama.

MiniCPM-o 4.5 Copy Chapter

MiniCPM-o 4.5 is the latest model in the MiniCPM-o series, a 9B end-to-end omnimodal model. It can continuously receive video and audio while generating speech and text, enabling real-time interactions that see, hear, and speak simultaneously.

Model highlights:

• 🔥 Vision capability approaching Gemini 2.5 Flash. It scores 77.6 on average across 8 popular OpenCompass benchmarks. With only 9B parameters, its visual understanding surpasses GPT-4o and Gemini 2.0 Pro, and a single model supports both instruct and thinking modes.
• 🎙 Strong speech capability. It supports bilingual real-time speech conversation with configurable voices in English and Chinese, making conversations more natural, expressive, and stable. A simple reference audio clip is enough for voice cloning and role play, and its voice cloning performance surpasses strong TTS tools such as CosyVoice2.
• 🎬 Full-duplex, proactive live interaction. It process real-time, continuous video and audio input streams simultaneously while generating concurrent text and speech output streams in an end-to-end fashion, without mutual blocking. This allows MiniCPM-o 4.5 to see, hear, and speak simultaneously, enabling smooth, real-time omnimodal conversations. Beyond responding to requests, the model can also interact proactively, for example by initiating reminders or comments based on its continuous understanding of the scene.

News Copy Section

• 2026-05-17 · Released API access for MiniCPM-o 4.5.

Model overview Copy Chapter

Get started Copy Chapter

Earlier updates Copy Chapter

• 2026-06-25 · MiniCPM-V 4.6 joined the official Ollama model library.
• 2026-05-17 · Released API access for MiniCPM-V 4.6.
• 2026-05-11 · Released MiniCPM-V 4.6, with mixed 4×/16× visual token compression and mobile deployment support.

Community milestones Copy Section

• 2025-01-19 · MiniCPM-o reached No. 1 on GitHub Trending and No. 2 on Hugging Face Trending.
• 2024-05-23 · MiniCPM-V reached No. 1 on both GitHub Trending and Hugging Face Trending.

Built with MiniCPM Copy Chapter

Community projects may support different model generations. Check each project's model list. More community projects

Project resources Copy Chapter

Developed by THUNLP and ModelBest. Code and model weights are released under Apache-2.0.

Citation Copy Chapter

If you find our model/code/paper helpful, please consider citing our papers 📝 and staring us ⭐️！

@misc{cui2026minicpmo45realtimefullduplex,
      title={MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction},
      author={Junbo Cui and Bokai Xu and Chongyi Wang and Tianyu Yu and Weiyue Sun and Yingjing Xu and Tianran Wang and Zhihui He and Wenshuo Ma and Tianchi Cai and others},
      year={2026},
      url={https://arxiv.org/abs/2604.27393},
}

@proceedings{yu2025minicpmv45cookingefficient,
      title={MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe},
      author={Tianyu Yu and Zefan Wang and Chongyi Wang and Fuwei Huang and Wenshuo Ma and Zhihui He and Tianchi Cai and Weize Chen and Yuxiang Huang and Yuanqian Zhao and others},
      year={2025},
      url={https://arxiv.org/abs/2509.18154},
}

@article{yao2024minicpm,
  title={MiniCPM-V: A GPT-4V Level MLLM on Your Phone},
  author={Yao, Yuan and Yu, Tianyu and Zhang, Ao and Wang, Chongyi and Cui, Junbo and Zhu, Hongji and Cai, Tianchi and Li, Haoyu and Zhao, Weilin and He, Zhihui and others},
  journal={arXiv preprint arXiv:2408.01800},
  year={2024}
}

Api (`docs/api.md`) Copy Section

API Guide

This document introduces how to use the Chat Completions APIs for MiniCPM-V 4.5 / 4.6 / 4.7 and MiniCPM-o 4.5, including text, image, and video understanding.

Endpoint Copy Chapter

Base URL: https://api.modelbest.cn/v1
Chat API: POST /chat/completions
Authorization: Bearer <API_KEY>
Content-Type: application/json

The API is OpenAI-compatible. Create an API key at platform.modelbest.cn, then:

export API_KEY="your-api-key"

Available Models Copy Chapter

MiniCPM-V-4.7-1B
MiniCPM-V-4.7-35B-A3B
MiniCPM-V-4.5-9B
MiniCPM-V-4.6-1B
MiniCPM-V-4.6-Thinking
MiniCPM-O-4.5-9B

MiniCPM-V 4.7 Copy Chapter

MiniCPM-V 4.7 comes in two sizes — MiniCPM-V-4.7-1B (dense, edge-friendly) and MiniCPM-V-4.7-35B-A3B (MoE, high-throughput). Both support text-only, image, and video understanding through the Chat Completions API. Pick the size by setting the model field.

Thinking is off by default; enable it per request with chat_template_kwargs (see below). The reasoning trace is returned in message.reasoning.

Text-Only Request Copy Section

curl https://api.modelbest.cn/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniCPM-V-4.7-1B",
    "messages": [
      {
        "role": "user",
        "content": "Introduce yourself in one sentence."
      }
    ]
  }'

For the MoE size, replace the model value with "MiniCPM-V-4.7-35B-A3B".

To enable thinking, add:

"chat_template_kwargs": {"enable_thinking": true}

The thinking model returns its intermediate reasoning in message.reasoning and the final answer in message.content.

Image Understanding Copy Section

Images are passed as base64 data URLs in the image_url content format.

curl https://api.modelbest.cn/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniCPM-V-4.7-1B",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Describe this image."
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "data:image/png;base64,<BASE64_IMAGE>"
            }
          }
        ]
      }
    ]
  }'

Video Understanding Copy Section

Videos are passed as base64 data URLs in the video_url content format (for example data:video/mp4;base64,...).

curl https://api.modelbest.cn/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniCPM-V-4.7-1B",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Describe this video briefly."
          },
          {
            "type": "video_url",
            "video_url": {
              "url": "data:video/mp4;base64,<BASE64_VIDEO>"
            }
          }
        ]
      }
    ]
  }'

MiniCPM-V 4.6 Copy Chapter

MiniCPM-V 4.6 can be called through the Chat Completions API. The interface supports text-only, image, and video understanding requests.

Text-Only Request Copy Section

curl https://api.modelbest.cn/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniCPM-V-4.6-1B",
    "messages": [
      {
        "role": "user",
        "content": "Introduce yourself in one sentence."
      }
    ]
  }'

To use the Thinking model, replace the model value with:

"MiniCPM-V-4.6-Thinking"

The Thinking model may return intermediate reasoning in message.reasoning.

Image Understanding Copy Section

Images are passed as base64 data URLs in the image_url content format.

curl https://api.modelbest.cn/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniCPM-V-4.6-1B",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Describe this image."
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "data:image/png;base64,<BASE64_IMAGE>"
            }
          }
        ]
      }
    ]
  }'

Video Understanding Copy Section

Videos are passed as base64 data URLs in the video_url content format (for example data:video/mp4;base64,...). The same format works for MiniCPM-V-4.7-1B, MiniCPM-V-4.7-35B-A3B, MiniCPM-V-4.5-9B, MiniCPM-V-4.6-1B, MiniCPM-V-4.6-Thinking, and MiniCPM-O-4.5-9B.

curl https://api.modelbest.cn/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniCPM-V-4.6-1B",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Describe this video briefly."
          },
          {
            "type": "video_url",
            "video_url": {
              "url": "data:video/mp4;base64,<BASE64_VIDEO>"
            }
          }
        ]
      }
    ]
  }'

Python Example Copy Section

import json
import os
import urllib.request

api_key = os.environ["API_KEY"]
payload = {
    "model": "MiniCPM-V-4.6-1B",
    "messages": [
        {
            "role": "user",
            "content": "List three use cases for MiniCPM-V.",
        }
    ],
}

request = urllib.request.Request(
    "https://api.modelbest.cn/v1/chat/completions",
    data=json.dumps(payload).encode("utf-8"),
    headers={
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
    },
    method="POST",
)

with urllib.request.urlopen(request) as response:
    data = json.loads(response.read().decode("utf-8"))

print(data["choices"][0]["message"]["content"])

Or with the OpenAI SDK:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["API_KEY"],
    base_url="https://api.modelbest.cn/v1",
)

resp = client.chat.completions.create(
    model="MiniCPM-V-4.6-1B",
    messages=[{"role": "user", "content": "List three use cases for MiniCPM-V."}],
)
print(resp.choices[0].message.content)

MiniCPM-V 4.5 Copy Chapter

Same Chat Completions API as above (text / image / video). Use model ID MiniCPM-V-4.5-9B:

curl https://api.modelbest.cn/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniCPM-V-4.5-9B",
    "messages": [
      {
        "role": "user",
        "content": "Introduce yourself in one sentence."
      }
    ]
  }'

MiniCPM-o 4.5 Copy Chapter

MiniCPM-o 4.5 can be called through the Chat Completions API for text, image, and video understanding requests, and also full-duplex realtime interaction.

Text-Only Request Copy Section

curl https://api.modelbest.cn/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniCPM-O-4.5-9B",
    "messages": [
      {
        "role": "user",
        "content": "Introduce yourself in one sentence."
      }
    ]
  }'

Image Understanding Copy Section

curl https://api.modelbest.cn/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniCPM-O-4.5-9B",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Describe this image."
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "data:image/png;base64,<BASE64_IMAGE>"
            }
          }
        ]
      }
    ]
  }'

Video Understanding Copy Section

curl https://api.modelbest.cn/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniCPM-O-4.5-9B",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Describe this video briefly."
          },
          {
            "type": "video_url",
            "video_url": {
              "url": "data:video/mp4;base64,<BASE64_VIDEO>"
            }
          }
        ]
      }
    ]
  }'

Realtime API Copy Section

MiniCPM-o 4.5 also supports full-duplex realtime multimodal interaction. For the Realtime API documentation, see the Realtime API Overview.

Best Practice Summary (`docs/best_practice_summary.md`) Copy Section

MiniCPM-V Best Practices

MiniCPM-V is a series of end-side multimodal LLMs (MLLMs) designed for vision-language understanding. The models take image, video and text as inputs and provide high-quality text output, aiming to achieve strong performance and efficient deployment. The most notable models in this series currently include MiniCPM-Llama3-V 2.5 and MiniCPM-V 2.6. The following sections provide detailed tutorials and guidance for each version of the MiniCPM-V models.

MiniCPM-V 2.6 Copy Chapter

MiniCPM-V 2.6 is the latest and most capable model in the MiniCPM-V series. With a total of 8B parameters, the model surpasses GPT-4V in single image, multi-image and video understanding. It outperforms GPT-4o mini, Gemini 1.5 Pro and Claude 3.5 Sonnet in single image understanding, and advances MiniCPM-Llama3-V 2.5's features such as strong OCR capability, trustworthy behavior, multilingual support, and end-side deployment. Due to its superior token density, MiniCPM-V 2.6 can for the first time support real-time video understanding on end-side devices such as iPad.

• Deployment Tutorial
• Training Tutorial
• Quantization Tutorial

MiniCPM-Llama3-V 2.5 Copy Chapter

MiniCPM-Llama3-V 2.5 is built on SigLip-400M and Llama3-8B-Instruct with a total of 8B parameters. It exhibits a significant performance improvement over MiniCPM-V 2.0.

• Quantization Tutorial
• Training Tutorial
• End-side Deployment
• Deployment Tutorial
• HD Decoding Tutorial
• Model Structure

Best Practice Summary Zh (`docs/best_practice_summary_zh.md`) Copy Section

MiniCPM-V 最佳实践

MiniCPM-V是面向图文理解的端侧多模态大模型系列。该系列模型接受图像和文本输入，并提供高质量的文本输出。自2024年2月以来，我们共发布了5个版本模型，旨在实现领先的性能和高效的部署，目前该系列最值得关注的模型包括：

MiniCPM-V 2.6 Copy Chapter

MiniCPM-V系列的最新、性能最佳模型。总参数量 8B，单图、多图和视频理解性能超越了 GPT-4V。在单图理解上，它取得了优于 GPT-4o mini、Gemini 1.5 Pro 和 Claude 3.5 Sonnet 等商用闭源模型的表现，并进一步优化了 MiniCPM-Llama3-V 2.5 的 OCR、可信行为、多语言支持以及端侧部署等诸多特性。基于其领先的视觉 token 密度，MiniCPM-V 2.6 成为了首个支持在 iPad 等端侧设备上进行实时视频理解的多模态大模型。

• 部署教程
• 训练教程
• 量化教程

MiniCPM-Llama3-V 2.5 Copy Chapter

MiniCPM-Llama3-V 2.5 基于 SigLip-400M 和 Llama3-8B-Instruct 构建，总共有 80 亿参数。其性能相比 MiniCPM-V 2.0 有了显著提升。

• 量化教程
• 训练教程
• 端侧部署
• 部署教程
• 高清解码教程
• 模型结构

Changelog En (`docs/changelog_en.md`) Copy Section

← Home · 中文

Changelog

Release announcements and project updates, newest first. Older entries describe support at the time of the announcement; use the current model and deployment guides for current instructions.

2026 Copy Chapter

• [2026.10.12] 🔥🔥🔥 We open-source MiniCPM-V 4.7, the first open-source MoE model to run smoothly on-device without compromising performance, setting a new multimodal performance–efficiency Pareto frontier. Try it now!

• [2026.06.25] 🔥🔥🔥 MiniCPM-V 4.6 has been merged into Ollama’s official model library, bringing ultra-efficient image and video understanding to local Ollama users. Try it now!

• [2026.05.17] ⭐️⭐️⭐️ We release the API service of MiniCPM-V 4.6 and MiniCPM-o 4.5.

• [2026.05.11] 🔥🔥🔥 We open-source MiniCPM-V 4.6, with mixed 4x/16x visual token compression. Powered by strong encoding efficiency and its lightweight 1.3B scale, it is our most edge-deployment-friendly model to date, achieving ~1.5x token throughput compared to Qwen3.5 0.8B. Try it now!

• [2026.05.07] 📢📢📢 We release the MiniCPM-o 4.5 technical report, introducing the key techniques behind its real-time full-duplex omni-modal interaction. Read it here.

• [2026.02.06] 🥳 🥳 🥳 We open-sourced a realtime web demo deployable on your own devices like Mac or GPU. Try it now!

• [2026.02.05] 📢📢📢 We note the web demo may experience latency issues due to network conditions. We are working actively to provide a Docker image for local deployment of the real-time interactive Demo as soon as possible. Please stay tuned!

• [2026.02.03] 🔥🔥🔥 We open-source MiniCPM-o 4.5, which matches Gemini 2.5 Flash on vision and speech, and supports full-duplex multimodal live streaming. Try it now!

2025 Copy Chapter

• [2025.09.18] 📢📢📢 MiniCPM-V 4.5 technical report is now released! See here.

• [2025.09.01] ⭐️⭐️⭐️ MiniCPM-V 4.5 has been officially supported by llama.cpp, vLLM, and LLaMA-Factory. You are welcome to use it directly through these official channels! Support for additional frameworks such as Ollama and SGLang is actively in progress.

• [2025.08.26] 🔥🔥🔥 We open-source MiniCPM-V 4.5, which outperforms GPT-4o-latest, Gemini-2.0 Pro, and Qwen2.5-VL 72B. It advances popular capabilities of MiniCPM-V, and brings useful new features. Try it now!

• [2025.08.02] 🚀🚀🚀 We open-source MiniCPM-V 4.0, which outperforms GPT-4.1-mini-20250414 in image understanding. It advances popular features of MiniCPM-V 2.6, and largely improves the efficiency. We also open-source the iOS App on iPhone and iPad. Try it now!

• [2025.08.01] ⭐️⭐️⭐️ We open-sourced the MiniCPM-V & o Cookbook! It provides comprehensive guides for diverse user scenarios, paired with our new Docs Site for smoother onboarding.

• [2025.06.20] ⭐️⭐️⭐️ Our official Ollama repository is released. Try our latest models with one click！

• [2025.03.01] 🚀🚀🚀 RLAIF-V, the alignment technique of MiniCPM-o, is accepted by CVPR 2025 Highlights！The code, dataset, paper are open-sourced!

• [2025.01.24] 📢📢📢 MiniCPM-o 2.6 technical report is released! See here.

• [2025.01.23] 💡💡💡 MiniCPM-o 2.6 is now supported by Align-Anything, a framework by PKU-Alignment Team for aligning any-to-any modality large models with human intentions. It supports DPO and SFT fine-tuning on both vision and audio. Try it now!

• [2025.01.19] ⭐️⭐️⭐️ MiniCPM-o tops GitHub Trending and reaches top-2 on Hugging Face Trending!

• [2025.01.19] 📢 ATTENTION! We are currently working on merging MiniCPM-o 2.6 into the official repositories of llama.cpp, Ollama, and vllm. Until the merge is complete, please USE OUR LOCAL FORKS of llama.cpp, Ollama, and vllm. Using the official repositories before the merge may lead to unexpected issues.

• [2025.01.17] We have updated the usage of MiniCPM-o 2.6 int4 quantization version and resolved the model initialization error. Click here and try it now!

• [2025.01.13] 🔥🔥🔥 We open-source MiniCPM-o 2.6, which matches GPT-4o-202405 on vision, speech and multimodal live streaming. It advances popular capabilities of MiniCPM-V 2.6, and supports various new fun features. Try it now!

2024 Copy Chapter

• [2024.08.17] 🚀🚀🚀 MiniCPM-V 2.6 is now fully supported by official llama.cpp! GGUF models of various sizes are available here.

• [2024.08.15] We now also support multi-image SFT. For more details, please refer to the document.

• [2024.08.14] MiniCPM-V 2.6 now also supports fine-tuning with the SWIFT framework!

• [2024.08.10] 🚀🚀🚀 MiniCPM-Llama3-V 2.5 is now fully supported by official llama.cpp! GGUF models of various sizes are available here.

• [2024.08.06] 🔥🔥🔥 We open-source MiniCPM-V 2.6, which outperforms GPT-4V on single image, multi-image and video understanding. It advances popular features of MiniCPM-Llama3-V 2.5, and can support real-time video understanding on iPad. Try it now!

• [2024.08.03] MiniCPM-Llama3-V 2.5 technical report is released! See here.

• [2024.07.19] MiniCPM-Llama3-V 2.5 supports vLLM now! See here.

• [2024.06.03] Now, you can run MiniCPM-Llama3-V 2.5 on multiple low VRAM GPUs(12 GB or 16 GB) by distributing the model's layers across multiple GPUs. For more details, check this link.

• [2024.05.28] 🚀🚀🚀 MiniCPM-Llama3-V 2.5 now fully supports its feature in llama.cpp and Ollama! Please pull the latest code of our provided forks (llama.cpp, Ollama). GGUF models in various sizes are available here. MiniCPM-Llama3-V 2.5 series is not supported by the official repositories yet, and we are working hard to merge PRs. Please stay tuned!

• [2024.05.28] 💫 We now support LoRA fine-tuning for MiniCPM-Llama3-V 2.5, using only 2 V100 GPUs! See more statistics here.

• [2024.05.25] MiniCPM-Llama3-V 2.5 now supports streaming outputs and customized system prompts. Try it here!

• [2024.05.24] We release the MiniCPM-Llama3-V 2.5 gguf, which supports llama.cpp inference and provides a 6~8 token/s smooth decoding on mobile phones. Try it now!

• [2024.05.23] 🔥🔥🔥 MiniCPM-V tops GitHub Trending and Hugging Face Trending! Our demo, recommended by Hugging Face Gradio’s official account, is available here. Come and try it out!

• [2024.05.23] 🔍 We've released a comprehensive comparison between Phi-3-vision-128k-instruct and MiniCPM-Llama3-V 2.5, including benchmark evaluations, multilingual capabilities, and inference efficiency 🌟📊🌍🚀. Click here to view more details.

• [2024.05.20] We open-soure MiniCPM-Llama3-V 2.5, it has improved OCR capability and supports 30+ languages, representing the first end-side MLLM achieving GPT-4V level performance! We provide efficient inference and simple fine-tuning. Try it now!

• [2024.04.23] MiniCPM-V-2.0 supports vLLM now! Click here to view more details.

• [2024.04.18] We create a HuggingFace Space to host the demo of MiniCPM-V 2.0 at here!

• [2024.04.17] MiniCPM-V-2.0 supports deploying WebUI Demo now!

• [2024.04.15] MiniCPM-V-2.0 now also supports fine-tuning with the SWIFT framework, with streaming inference enabled!

• [2024.04.12] We open-source MiniCPM-V 2.0, which achieves comparable performance with Gemini Pro in understanding scene text and outperforms strong Qwen-VL-Chat 9.6B and Yi-VL 34B on OpenCompass, a comprehensive evaluation over 11 popular benchmarks. Click here to view the MiniCPM-V 2.0 technical blog.

• [2024.03.14] MiniCPM-V now supports fine-tuning with the SWIFT framework. Thanks to Jintao for the contribution！

• [2024.03.01] MiniCPM-V can now be deployed on Mac!

• [2024.02.01] We open-source MiniCPM-V and OmniLMM-12B, which support efficient end-side deployment and powerful multimodal capabilities correspondingly.

Changelog Zh (`docs/changelog_zh.md`) Copy Section

← 首页 · English

更新日志

按日期倒序保留模型发布与项目更新。历史消息中的框架支持状态对应消息发布时点；当前使用方式请以模型与部署文档为准。

2026 Copy Chapter

• [2026.10.12] 🔥🔥🔥 我们开源了 MiniCPM-V 4.7，首个端侧无损流畅运行的开源 MoE 模型，刷新多模态性能—效率帕累托前沿。欢迎试用！

• [2026.06.25] 🔥🔥🔥 MiniCPM-V 4.6 已正式合并至 Ollama 官方模型库，为 Ollama 用户带来高效的图像与视频理解能力。欢迎体验！

• [2026.05.17] ⭐️⭐️⭐️ 我们发布了 MiniCPM-V 4.6 和 MiniCPM-o 4.5 的 API 接口。

• [2026.05.11] 🔥🔥🔥 我们开源了 MiniCPM-V 4.6，支持 4倍/16倍 混合视觉 token 压缩率，凭借出色的编码效率和 1.3B 的轻量规模，它是我们端侧部署最友好的一代模型，高并发场景 token 吞吐达到 Qwen3.5 0.8B 的 ~1.5 倍。欢迎试用！

• [2026.05.07] 📢📢📢 我们发布了 MiniCPM-o 4.5 技术报告，介绍了其实现实时全双工全模态交互的关键技术。欢迎点击这里查看。

• [2026.02.06] 🥳 🥳 🥳 我们开源了可在 Mac 或 GPU 等本地设备上部署的实时 Web Demo。立即体验！

• [2026.02.05] 📢📢📢 我们注意到，由于网络状况原因，网页版演示可能会出现显著的延迟问题。我们正在积极工作，将尽快提供实时交互演示版的Docker镜像供本地部署，敬请持续关注！

• [2026.02.03] 🔥🔥🔥 我们开源了 MiniCPM-o 4.5，该模型视觉和语音能力达到了 Gemini 2.5 Flash 水平，同时支持全双工多模态流式交互。欢迎试用！

2025 Copy Chapter

• [2025.09.18] 📢📢📢 MiniCPM-V 4.5 技术报告已发布! 欢迎点击这里查看.

• [2025.09.01] ⭐️⭐️⭐️ MiniCPM-V 4.5 已被 llama.cpp、vLLM 和 LLaMA-Factory 等框架官方支持，欢迎从官方入口直接使用！更多框架如 Ollama 和 SGLang 的官方支持正在持续适配中！

• [2025.08.26] 🔥🔥🔥 我们开源了 MiniCPM-V 4.5，其视觉性能超越了 GPT-4o-latest、Gemini-2.0 Pro 和 Qwen2.5-VL 72B。它不仅延续并强化了 MiniCPM-V 的热门能力，还带来了诸多实用的新功能。欢迎试用！

• [2025.08.02] 🚀🚀🚀 我们开源了 MiniCPM-V 4.0，该模型在图像理解能力上超越了 GPT-4.1-mini-20250414。该模型不仅继承了 MiniCPM-V 2.6 的众多实用特性，还大幅提升了推理效率。我们还同步开源了适用于 iPhone 和 iPad 的 iOS 应用，欢迎试用！

• [2025.08.01] ⭐️⭐️⭐️ 我们开源了 MiniCPM-V & o Cookbook，提供针对不同人群的全场景使用指南，配合最新的文档网站上手更轻松！

• [2025.06.20] ⭐️⭐️⭐️ MiniCPM-o 的 Ollama 官方仓库正式支持 MiniCPM-o 2.6 等模型啦，欢迎一键使用！

• [2025.03.01] 🚀🚀🚀 MiniCPM-o 系列的对齐技术 RLAIF-V 被 CVPR 2025 接收为 Highlights 了！其代码、数据、论文均已开源。

• [2025.01.24] 📢📢📢 MiniCPM-o 2.6 技术报告已发布! 欢迎点击这里查看.

• [2025.01.23] 💡💡💡 MiniCPM-o 2.6 现在已被北大团队开发的 Align-Anything，一个用于对齐全模态大模型的框架集成，支持 DPO 和 SFT 在视觉和音频模态上的微调。欢迎试用！

• [2025.01.19] ⭐️⭐️⭐️ MiniCPM-o 在 GitHub Trending 上登顶， Hugging Face Trending 上也达到了第二！

• [2025.01.19] 📢 注意! 我们正在努力将 MiniCPM-o 2.6 的支持合并到 llama.cpp、Ollama、vLLM 的官方仓库，但还未完成。请大家暂时先使用我们提供的 fork 来进行部署：llama.cpp、Ollama、vllm。 合并完成前，使用官方仓库可能会导致不可预期的问题。

• [2025.01.17] 我们更新了 MiniCPM-o 2.6 int4 量化版本的使用方式，解决了模型初始化的问题，欢迎点击这里试用！

• [2025.01.13] 🔥🔥🔥 我们开源了 MiniCPM-o 2.6，该模型视觉、语音和多模态流式能力达到了 GPT-4o-202405 级别，进一步优化了 MiniCPM-V 2.6 的众多亮点能力，还支持了很多有趣的新功能。欢迎试用！

2024 Copy Chapter

• [2024.08.17] 🚀🚀🚀 llama.cpp 官方仓库正式支持 MiniCPM-V 2.6 啦！点击这里查看各种大小的 GGUF 版本。

• [2024.08.15] MiniCPM-V 2.6 现在支持多图像 SFT。有关更多详细信息，请参阅微调文档

• [2024.08.14] MiniCPM-V 2.6 现在可以通过 SWIFT 框架 微调 了！

• [2024.08.10] 🚀🚀🚀 llama.cpp 官方仓库正式支持 MiniCPM-Llama3-V 2.5 啦！点击这里查看各种大小的 GGUF 版本。

• [2024.08.06] 🔥🔥🔥 我们开源了 MiniCPM-V 2.6，该模型在单图、多图和视频理解方面取得了优于 GPT-4V 的表现。我们还进一步提升了 MiniCPM-Llama3-V 2.5 的多项亮点能力，并首次支持了 iPad 上的实时视频理解。欢迎试用！

• [2024.08.03] MiniCPM-Llama3-V 2.5 技术报告已发布！欢迎点击这里查看。

• [2024.07.19] MiniCPM-Llama3-V 2.5 现已支持vLLM ！

• [2024.06.03] 现在，你可以利用多张低显存显卡（12G/16G）进行GPU串行推理。详情请参见该文档配置。

• [2024.05.28] 💫 我们现在支持 MiniCPM-Llama3-V 2.5 的 LoRA 微调，更多内存使用统计信息可以在这里找到。

• [2024.05.28] 💥 MiniCPM-Llama3-V 2.5 现在在 llama.cpp 和 Ollama 中完全支持其功能！请拉取我们最新的 fork 来使用：llama.cpp & ollama。我们还发布了各种大小的 GGUF 版本，请点击这里查看。请注意，目前官方仓库尚未支持 MiniCPM-Llama3-V 2.5，我们也正积极推进将这些功能合并到 llama.cpp & ollama 官方仓库，敬请关注！

• [2024.05.25] MiniCPM-Llama3-V 2.5 支持流式输出和自定义系统提示词了，欢迎试用!

• [2024.05.24] 我们开源了 MiniCPM-Llama3-V 2.5 gguf，支持 llama.cpp 推理！实现端侧 6-8 tokens/s 的流畅解码，欢迎试用！

• [2024.05.23] 🔥🔥🔥 MiniCPM-V 在 GitHub Trending 和 Hugging Face Trending 上登顶！MiniCPM-Llama3-V 2.5 Demo 被 Hugging Face 的 Gradio 官方账户推荐，欢迎点击这里体验！

• [2024.05.23] 🔍 我们添加了Phi-3-vision-128k-instruct 与 MiniCPM-Llama3-V 2.5的全面对比，包括基准测试评估、多语言能力和推理效率 🌟📊🌍🚀。点击这里查看详细信息。

• [2024.05.20] 我们开源了 MiniCPM-Llama3-V 2.5，增强了 OCR 能力，支持 30 多种语言，并首次在端侧实现了 GPT-4V 级的多模态能力！我们提供了高效推理和简易微调的支持，欢迎试用！

• [2024.04.23] 我们增加了MiniCPM-V 2.0对 vLLM 的支持，欢迎体验！

• [2024.04.18] 我们在 HuggingFace Space 新增了 MiniCPM-V 2.0 的 demo，欢迎体验！

• [2024.04.17] MiniCPM-V 2.0 现在支持用户部署本地 WebUI Demo 了，欢迎试用!

• [2024.04.15] MiniCPM-V 2.0 现在可以通过 SWIFT 框架 微调 了，支持流式输出!

• [2024.04.12] 我们开源了 MiniCPM-V 2.0，该模型刷新了 OCRBench 开源模型最佳成绩，在场景文字识别能力上比肩 Gemini Pro，同时还在综合了 11 个主流多模态大模型评测基准的 OpenCompass 榜单上超过了 Qwen-VL-Chat 9.6B、CogVLM-Chat 17B 和 Yi-VL 34B 等更大参数规模的模型！点击 这里 查看 MiniCPM-V 2.0 技术博客。

• [2024.03.14] MiniCPM-V 现在支持 SWIFT 框架下的微调了，感谢 Jintao 的贡献！

• [2024.03.01] MiniCPM-V 现在支持在 Mac 电脑上进行部署！

• [2024.02.01] 我们开源了 MiniCPM-V 和 OmniLMM-12B，分别可以支持高效的端侧部署和同规模领先的多模态能力！

Compare With Phi 3 Vision (`docs/compare_with_phi-3_vision.md`) Copy Section

Phi-3-vision-128K-Instruct vs MiniCPM-Llama3-V 2.5 Copy Chapter

Comparison results of Phi-3-vision-128K-Instruct and MiniCPM-Llama3-V 2.5, regarding the model size, hardware requirements, and performances.
With int4 quantization, MiniCPM-Llama3-V 2.5 delivers smooth inference with only 8GB of GPU memory. In most benchmarks, MiniCPM-Llama3-V 2.5 achieves better performance compared with Phi-3-vision-128K-Instruct. Moreover, MiniCPM-Llama3-V 2.5 also exhibits lower latency and better throughtput even without quantization.

我们提供了从模型参数、硬件需求、性能指标等方面对比 Phi-3-vision-128K-Instruct 和 MiniCPM-Llama3-V 2.5 的结果。通过 int4 量化，MiniCPM-Llama3-V 2.5 仅需 8GB 显存即可推理。在大多数评测集上， MiniCPM-Llama3-V 2.5 相比于 Phi-3-vision-128K-Instruct 都展现出了更优的性能表现。 即使未经量化，MiniCPM-Llama3-V 2.5 的推理延迟和吞吐率也都更具优势。

Multilingual Capabilities（多语言能力对比） Copy Section

MiniCPM-Llama3-V 2.5 exhibits stronger multilingual capabilities compared with Phi-3-vision-128K-Instruct on LLaVA Bench.

MiniCPM-Llama3-V 2.5 在对话和推理评测榜单 LLaVA Bench 上展现出了比 Phi-3-vision-128K-Instruct 更强的多语言的性能。

Deployment En (`docs/deployment_en.md`) Copy Section

← Home · 中文

Deployment, Training, and API

Run the code examples from the repository root unless stated otherwise.

MiniCPM-V & o Cookbook Copy Chapter

The MiniCPM-V & o Cookbook provides scenario-based recipes for deploying, fine-tuning, quantizing, and building demos with MiniCPM-V and MiniCPM-o. The documentation website presents these recipes in a structured way for quick lookup.

For MiniCPM-o 4.5 API usage, see the API Guide. Previous MiniCPM-V 4.6 usage is available in its archived documentation.

It is organized for:

• Individuals: Local inference, quantized deployment, and end-device demos.
• Enterprises: Scalable serving, high-throughput inference, and production-oriented deployment.
• Researchers: Fine-tuning, model adaptation, and experimental workflows.

For framework-specific deployment and training guides, see Supported Inference and Training Frameworks and API.

Supported Inference and Training Frameworks and API Copy Chapter

Inference: vLLM, SGLang, llama.cpp, Ollama, FlagOS Copy Section

We support inference with vLLM, SGLang, llama.cpp and Ollama. See the per-model deployment guides below, or refer to our Cookbook for more details.

FlagOS

FlagOS supports inference of MiniCPM-o 4.5 across six AI chip families, including Nvidia GPUs.

Click to show FlagOS support details.

To enable large-scale deployment across different AI chips, Beijing Zhiyuan Research Institute, together with numerous research institutions, chip manufacturers, system vendors, and algorithm and software organizations both domestically and internationally, jointly initiated and established the FlagOS Open Source Community.

The FlagOS community is dedicated to building a unified, open-source system software stack for various AI chips, encompassing core open-source projects such as a large-scale operator library, a unified AI compiler, parallel training and inference frameworks, and a unified communication library. It aims to create an open technology ecosystem connecting the “model-system-chip” layers. By enabling “develop once, deploy across chips”, FlagOS unlocks the computational potential of hardware, breaks down the ecosystem silos between different chip software stacks, and effectively reduces migration costs for developers. The FlagOS community fosters an AI hardware and software ecosystem, overcomes single-vendor closed-source monopolies, promotes widespread deployment of AI hardware technologies, and is committed to rooted in China while embracing global collaboration.
Official website: https://flagos.io.

FlagOS: Supporting Multiple AI Chips

Thanks to FlagOS’s unified multi-chip AI system software stack, MiniCPM-o 4.5 was adapted to 6 different AI chips in an extremely short time. Currently, the multi-chip version of MiniCPM-o 4.5 has been released on FlagRelease, FlagOS’s platform for automatic migration, adaptation, and deployment of large models across multi-architecture AI chips. Details are as follows:

Comprehensive Evaluation

Transformers-FlagOS version

Accuracy Difference between USE_FLAGOS=1 on multi-backend and USE_FLAGOS=0 on Nvidia-CUDA

VLLM-FlagOS version

Accuracy Difference between USE_FLAGGEMS=1 FLAGCX_PATH=/workspace/FlagCX on Nvidia or USE_FLAGGEMS=1 on ZHENW 810E, and launching vllm server directly on Nvidia

FlagOS Usage

FlagOS Performance Acceleration on Nvidia

On the Transformers version, under the premise of precision alignment between the CUDA and FlagOS ecosystems, FlagOS achieves a 6% performance improvement in total task execution time compared to CUDA.

From FlagRelease【Recommendation】

FlagRelease is a platform developed by the FlagOS team for automatic migration, adaptation, and deployment of large models across multi-architecture AI chips. The multi-chip version of MiniCPM-o 4.5 has already been released on FlagRelease. All necessary software packages are pre-installed on the platform, so users do not need to install anything.

• FlagRelease Image Key Versions

  Component	Version
  Accelerator Card Driver	570.158.01
  CUDA SDK Build	cuda_13.0.r13.0/compiler.36424714_0
  FlagTree	0.4.0+3.5
  FlagGems	4.2.1rc0
  vllm & vllm-plugin-fl	0.13.0 + vllm_fl 0.0.0
  FlagCX	0.1.0

• FlagRelease Quick Start

  Vendor	ModelScope	Huggingface
  Nvidia	MiniCPM-o-4.5-nvidia-FlagOS	MiniCPM-o-4.5-nvidia-FlagOS
  Hygon-BW1000	MiniCPM-o-4.5-hygon-FlagOS	MiniCPM-o-4.5-hygon-FlagOS
  Metax-C550	MiniCPM-o-4.5-metax-FlagOS	MiniCPM-o-4.5-metax-FlagOS
  Iluvatar-BIV150	MiniCPM-o-4.5-iluvatar-FlagOS	MiniCPM-o-4.5-iluvatar-FlagOS
  Ascend-A3	MiniCPM-o-4.5-ascend-FlagOS	MiniCPM-o-4.5-ascend-FlagOS
  Zhenwu-810E	MiniCPM-o-4.5-zhenwu-FlagOS	MiniCPM-o-4.5-zhenwu-FlagOS

From Scratch

• Dependencies: Python 3.12, GLIBC 2.39, GLIBCXX 3.4.33, CXXABI 1.3.15

Transformers

• Installing the FlagOS Operator Library

  Official Repository: https://github.com/flagos-ai/FlagGems

  shell Copy

  pip install flag-gems==4.2.1rc0

• Installing the FlagOS Compiler

  Official Repository: https://github.com/flagos-ai/flagtree

  Quick Reference for Core Dependency Versions: https://github.com/flagos-ai/FlagTree/blob/main/documents/build.md#tips-for-building

  shell Copy

  pip uninstall triton
  
  python3 -m pip install flagtree==0.4.0+3.5 --index-url=https://resource.flagos.net/repository/flagos-pypi-hosted/simple --trusted-host=https://resource.flagos.net

• Activating Acceleration

  Add USE_FLAGOS=1 before the command for the task you want to run. For example, when you run:

  shell Copy

  python3 generate_speech_from_video.py

  To use the MiniCPM-o-4.5 model to generate spoken responses from video content, you can:

  shell Copy

  USE_FLAGOS=1 python3 generate_speech_from_video.py

  to accelerate this process with FlagOS.

Vllm Version

• Installing the FlagOS Operator Library

  Official Repository: https://github.com/flagos-ai/FlagGems

  shell Copy

  pip install flag-gems==4.2.1rc0
  pip install triton==3.5.1

• Activating Acceleration

  Add USE_FLAGOS=1 before the command for the task you want to run. For example, when you run:

  shell Copy

  vllm serve ${model_path} --dtype auto  --gpu_memory_utilization 0.9 --trust-remote-code --max-num-batched-tokens 2048 --served-model-name cpmo --port ${Port}

  To start the MiniCPM-o-4.5 server, you can:

  shell Copy

  USE_FLAGOS=1 vllm serve ${model_path} --dtype auto  --gpu_memory_utilization 0.9 --trust-remote-code --max-num-batched-tokens 2048 --served-model-name cpmo --port ${Port}

  to accelerate this process with FlagOS.

Using FlagOS Unified Multi-Chip Backend Plugin

vllm-plugin-FL is a plugin built for the vLLM inference/service framework. Developed on top of FlagOS’s unified multi-chip backend, it is designed to extend vLLM’s capabilities and performance across a variety of hardware environments.

Using vllm-plugin-FL

Training: LLaMA-Factory, SWIFT Copy Section

We support fine-tuning with LLaMA-Factory, SWIFT. Refer to our Cookbook for more details.

API Copy Section

Refer to the API Guide for MiniCPM-V 4.7 / 4.5 and MiniCPM-o 4.5. The MiniCPM-V 4.6 API entry is preserved in its archived documentation.

Deployment Zh (`docs/deployment_zh.md`) Copy Section

← 首页 · English

部署、训练与 API

除另有说明外，以下代码均从仓库根目录运行。

MiniCPM-V & o 使用手册 Copy Chapter

欢迎探索我们整理的 MiniCPM-V & o 使用手册 (Cookbook)。Cookbook 提供面向场景的教程，覆盖 MiniCPM-V 和 MiniCPM-o 的部署、微调、量化和 Demo 构建等常见任务；配套的文档网站以结构化方式呈现这些方案，便于快速查找。

MiniCPM-o 4.5 的 API 用法见 API Guide。历史版本 MiniCPM-V 4.6 的用法见独立文档。

它面向以下使用场景组织内容：

• 个人用户：本地推理、量化部署和端侧 Demo。
• 企业用户：可扩展服务、高吞吐推理和生产部署。
• 研究者：微调、模型适配和实验工作流。

如需查看具体框架的部署和训练指南，请参考训练和推理框架与 API 支持。

训练和推理框架与 API 支持 Copy Chapter

推理：vLLM、SGLang、llama.cpp、Ollama、FlagOS Copy Section

MiniCPM-V 和 MiniCPM-o 模型推理适配 vLLM、SGLang、llama.cpp、Ollama 等框架。可点击下表查看各模型的部署指南，或参考我们的使用指南。

FlagOS

FlagOS 平台支持 MiniCPM-o 4.5 在6 种不同的 AI 芯片上的推理，包括 Nvidia GPU 等。

点击展开 FlagOS 上的 MiniCPM-o 4.5 推理详细

为解决不同 AI 芯片大规模落地应用，北京智源研究院联合众多科研机构、芯片企业、系统厂商、算法和软件相关单位等国内外机构共同发起并创立了 FlagOS 开源社区。

FlagOS 社区致力于打造面向多种 AI 芯片的统一、开源的系统软件栈，包括大型算子库、统一AI编译器、并行训推框架、统一通信库等核心开源项目，构建「模型-系统-芯片」三层贯通的开放技术生态，通过“一次开发跨芯迁移”释放硬件计算潜力，打破不同芯片软件栈之间生态隔离，有效降低开发者的迁移成本。FlagOS 社区构建人工智能软硬件生态，突破单一闭源垄断，推动AI硬件技术大范围落地发展，立足中国、拥抱全球合作。
官网速递：https://flagos.io

FlagOS 多 AI 芯片支持

基于FlagOS极短时间内适配MiniCPM-o 4.5到 6 种不同的 AI 芯片，得益于众智 FlagOS 的多芯片统一 AI 系统软件栈的能力。目前，在FlagOS团队构建的面向多架构人工智能芯片的大模型自动迁移、适配与发布平台FlagRelease上，已发布MiniCPM-o-4.5的多芯片版本。细节如下：

综合评估

Transformers–FlagOS 版本

在多后端环境下使用 USE_FLAGOS=1 与在 NVIDIA CUDA 上使用 USE_FLAGOS=0 的精度差异

VLLM-FlagOS 版本

在 NVIDIA 上使用 USE_FLAGGEMS=1 FLAGCX_PATH=/workspace/FlagCX，或在真武 810E USE_FLAGGEMS=1，与直接在 NVIDIA 平台上启动 vLLM Server 的精度差异

FlagOS 使用方式

使用 FlagOS 在Nvidia体验性能加速

在Transformers版本上，CUDA生态与FlagOS生态精度对齐的前提下，FlagOS相比CUDA任务的负载执行总时间有6%的性能提升。

From FlagRelease【推荐】

FlagRelease是FlagOS团队构建的一套面向多架构人工智能芯片的大模型自动迁移、适配与发布平台，已发布MiniCPM-o-4.5的多芯片版本。FlagRelase已内置相关软件包，无需用户安装。

• FlagRelease 镜像关键版本信息

  组件	版本
  加速卡驱动	570.158.01
  CUDA SDK Build	cuda_13.0.r13.0/compiler.36424714_0
  FlagTree	0.4.0+3.5
  FlagGems	4.2.1rc0
  vllm & vllm-plugin-fl	0.13.0 + vllm_fl 0.0.0
  FlagCX	0.1.0

• FlagRelease 使用速递

  Vendor	ModelScope	Huggingface
  Nvidia	MiniCPM-o-4.5-nvidia-FlagOS	MiniCPM-o-4.5-nvidia-FlagOS
  Hygon-BW1000	MiniCPM-o-4.5-hygon-FlagOS	MiniCPM-o-4.5-hygon-FlagOS
  Metax-C550	MiniCPM-o-4.5-metax-FlagOS	MiniCPM-o-4.5-metax-FlagOS
  Iluvatar-BIV150	MiniCPM-o-4.5-iluvatar-FlagOS	MiniCPM-o-4.5-iluvatar-FlagOS
  Ascend-A3	MiniCPM-o-4.5-ascend-FlagOS	MiniCPM-o-4.5-ascend-FlagOS
  Zhenwu-810E	MiniCPM-o-4.5-zhenwu-FlagOS	MiniCPM-o-4.5-zhenwu-FlagOS

从零开始

• 依赖Python3.12, GLIBC_2.39, GLIBCXX_3.4.33, CXXABI_1.3.15 环境

Transformers 版本

• 安装FlagOS算子库

  官方仓库：https://github.com/flagos-ai/FlagGems

  shell Copy

  pip install flag-gems==4.2.1rc0

• 安装FlagOS编译器

  官方仓库：https://github.com/flagos-ai/flagtree

  底层依赖库版本速查：https://github.com/flagos-ai/FlagTree/blob/main/documents/build.md#tips-for-building

  shell Copy

  pip uninstall triton
  
  python3 -m pip install flagtree==0.4.0+3.5 --index-url=https://resource.flagos.net/repository/flagos-pypi-hosted/simple --trusted-host=https://resource.flagos.net

• 开启加速

  在欲执行任务的命令前添加 USE_FLAGOS=1，例如，当您通过

  shell Copy

  python3 generate_speech_from_video.py

  来使用 MiniCPM-o-4.5 模型根据视频内容生成语音回答时，可以通过

  shell Copy

  USE_FLAGOS=1 python3 generate_speech_from_video.py

  来使用 FlagOS 加速这一过程。

Vllm 版本

• 安装FlagOS算子库

  官方仓库：https://github.com/flagos-ai/FlagGems

  shell Copy

  pip install flag-gems==4.2.1rc0
  pip install triton==3.5.1

• 开启加速
  在欲执行任务的命令前添加 USE_FLAGOS=1，例如，当您通过

  shell Copy

  vllm serve ${model_path} --dtype auto  --gpu_memory_utilization 0.9 --trust-remote-code --max-num-batched-tokens 2048 --served-model-name cpmo --port ${Port}

  来启动MiniCPM-o-4.5服务端时，可以通过

  shell Copy

  USE_FLAGOS=1 vllm serve ${model_path} --dtype auto  --gpu_memory_utilization 0.9 --trust-remote-code --max-num-batched-tokens 2048 --served-model-name cpmo --port ${Port}

  来使用FlagOS加速这一过程。

使用 FlagOS 统一多芯片后端插件

vllm-plugin-FL 是一个为 vLLM 推理/服务框架构建的插件，它基于 FlagOS 的统一多芯片后端 开发，旨在扩展 vLLM 在多种硬件环境下的功能和性能表现。

Using vllm-plugin-FL

训练：LLaMA-Factory、SWIFT Copy Section

MiniCPM-V 和 MiniCPM-o 模型支持通过 LLaMA-Factory 和 SWIFT 等框架训练。具体用法可以参考我们的使用指南。

API Copy Section

MiniCPM-V 4.7 / 4.5 和 MiniCPM-o 4.5 的 API 用法见 API Guide。MiniCPM-V 4.6 的 API 入口已保存在独立文档中。

Faqs (`docs/faqs.md`) Copy Section

FAQs Copy Section

Q: How to choose between sampling or beam search for inference

In various scenarios, the quality of results obtained from beam search and sampling decoding strategies can vary. You can determine your decoding strategy based on the following aspects:

If you have the following needs, consider using sampling decoding:

1. You require faster inference speed.
2. You wish for a streaming generation approach.
3. Your task necessitates some open-ended responses.

If your task is about providing deterministic answers, you might want to experiment with beam search to see if it can achieve better outcomes.

Q: How to ensure that the model generates results of sufficient length

We've observed that during multi-language inference on MiniCPM-V 2.6, the generation sometimes ends prematurely. You can improve the results by passing a min_new_tokens parameter.

res = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer,
    min_new_tokens=100
)

Inference On Multiple Gpus (`docs/inference_on_multiple_gpus.md`) Copy Section

Using MiniCPM-Llama3-V-2_5 with Multiple GPUs Copy Chapter

Due to the limited memory capacity of a single GPU, it may be impossible to load the entire MiniCPMV model (the model weights account for 18 GiB) onto one device for inference (assume one gpu only has 12 GiB or 16 GiB GPU memory). To address this limitation, multi-GPU inference can be employed, where the model's layers are distributed across multiple GPUs.

A minimal modification method can be used to achieve this distribution, ensuring that the layers are assigned to different GPUs with minimal changes to the original model structure.

To implement this, we utilize features provided by accelerate library.

Install all requirements of MiniCPM-Llama3-V-2_5, additionally, you also need to install accelerate.

pip install accelerate

Example Usage for `2x16GiB` GPUs Copy Section

Now we consider a demo for two GPUs, where each GPU has 16 GiB GPU memory.

1. Import necessary libraries.

from PIL import Image
import torch
from transformers import AutoConfig, AutoModel, AutoTokenizer
from accelerate import init_empty_weights, infer_auto_device_map, load_checkpoint_in_model, dispatch_model

2. Download model weights.

MODEL_PATH = '/local/path/to/MiniCPM-Llama3-V-2_5' # you can download in advance or use `openbmb/MiniCPM-Llama3-V-2_5`

3. Determine the distribution of layers on multiple GPUs.

max_memory_each_gpu = '10GiB' # Define the maximum memory to use on each gpu, here we suggest using a balanced value, because the weight is not everything, the intermediate activation value also uses GPU memory (10GiB < 16GiB)

gpu_device_ids = [0, 1] # Define which gpu to use (now we have two GPUs, each has 16GiB memory)

no_split_module_classes = ["LlamaDecoderLayer"]

max_memory = {
    device_id: max_memory_each_gpu for device_id in gpu_device_ids
}

config = AutoConfig.from_pretrained(
    MODEL_PATH, 
    trust_remote_code=True
)

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_PATH, 
    trust_remote_code=True
)

with init_empty_weights():
    model = AutoModel.from_config(
        config, 
        torch_dtype=torch.float16, 
        trust_remote_code=True
    )

device_map = infer_auto_device_map(
    model,
    max_memory=max_memory, no_split_module_classes=no_split_module_classes
)

print("auto determined device_map", device_map)

# Here we want to make sure the input and output layer are all on the first gpu to avoid any modifications to original inference script.

device_map["llm.model.embed_tokens"] = 0
device_map["llm.model.layers.0"] = 0
device_map["llm.lm_head"] = 0
device_map["vpm"] = 0
device_map["resampler"] = 0

print("modified device_map", device_map)

You may see this output:

modified device_map OrderedDict([('llm.model.embed_tokens', 0), ('llm.model.layers.0', 0), ('llm.model.layers.1', 0), ('llm.model.layers.2', 0), ('llm.model.layers.3', 0), ('llm.model.layers.4', 0), ('llm.model.layers.5', 0), ('llm.model.layers.6', 0), ('llm.model.layers.7', 0), ('llm.model.layers.8', 0), ('llm.model.layers.9', 0), ('llm.model.layers.10', 0), ('llm.model.layers.11', 0), ('llm.model.layers.12', 0), ('llm.model.layers.13', 0), ('llm.model.layers.14', 0), ('llm.model.layers.15', 0), ('llm.model.layers.16', 1), ('llm.model.layers.17', 1), ('llm.model.layers.18', 1), ('llm.model.layers.19', 1), ('llm.model.layers.20', 1), ('llm.model.layers.21', 1), ('llm.model.layers.22', 1), ('llm.model.layers.23', 1), ('llm.model.layers.24', 1), ('llm.model.layers.25', 1), ('llm.model.layers.26', 1), ('llm.model.layers.27', 1), ('llm.model.layers.28', 1), ('llm.model.layers.29', 1), ('llm.model.layers.30', 1), ('llm.model.layers.31', 1), ('llm.model.norm', 1), ('llm.lm_head', 0), ('vpm', 0), ('resampler', 0)])

4. Next, use the device_map to dispatch the model layers to corresponding gpus.

load_checkpoint_in_model(
    model, 
    MODEL_PATH, 
    device_map=device_map)

model = dispatch_model(
    model, 
    device_map=device_map
)

torch.set_grad_enabled(False)

model.eval()

5. Chat!

image_path = '/local/path/to/test.png'

response = model.chat(
    image=Image.open(image_path).convert("RGB"),
    msgs=[
        {
            "role": "user",
            "content": "guess what I am doing?"
        }
    ],
    tokenizer=tokenizer
)

print(response)

In this case the OOM (CUDA out of memory) problem may be eliminated. We have tested that:

• it works well for 3000 text input tokens and 1000 text output tokens.
• it works well for a high-resolution input image.

Usage for general cases Copy Section

It is similar to the previous example, but you may consider modifying these two variables.

max_memory_each_gpu = '10GiB' # Define the maximum memory to use on each gpu, here we suggest using a balanced value, because the weight is not everything, the intermediate activation value also uses GPU memory (10GiB < 16GiB)

gpu_device_ids = [0, 1, ...] # Define which gpu to use (now we have two GPUs, each has 16GiB memory)

You may use the following shell script to monitor the memory usage during inference, if there is OOM, try to reduce max_memory_each_gpu, to make memory pressure more balanced across all gpus.

watch -n1 nvidia-smi

References Copy Section

Ref 1

Llamafactory Train And Infer (`docs/llamafactory_train_and_infer.md`) Copy Section

Best Practice with LLaMA-Factory

Contents Copy Chapter

• Support Models
• LLaMA-Factory Installation
• Dataset Prepare
  • Image Dataset
  • Video Dataset
  • Audio Dataset
• Lora Fine-Tuning
• Full Parameters Fine-Tuning
• Inference

Support Models Copy Chapter

• openbmb/MiniCPM-V-4
• openbmb/MiniCPM-o-2_6
• openbmb/MiniCPM-V-2_6

LLaMA-Factory Installation Copy Chapter

You can install LLaMA-Factory using commands below.

git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics,deepspeed,minicpm_v]"
mkdir configs # let's put all yaml files here

Dataset Prepare Copy Chapter

Refer to data/dataset_info.json to add your customised dataset. Let's use the two existing demo datasets mllm_demo, mllm_video_demo and mllm_audio_demo as examples (audio is only for MiniCPM-o-2.6).

Image Dataset Copy Section

Refer to image sft demo data: data/mllm_demo.json

**data/mllm_demo.json**

/* Detailed source-code truncated for AI context efficiency. */

Video Dataset Copy Section

Refer to video sft demo data: data/mllm_video_demo.json

**data/mllm_video_demo.json**

[
  {
    "messages": [
      {
        "content": "<video>Why is this video funny?",
        "role": "user"
      },
      {
        "content": "Because a baby is reading, and he is so cute!",
        "role": "assistant"
      }
    ],
    "videos": [
      "mllm_demo_data/1.mp4"
    ]
  },
  {
    "messages": [
      {
        "content": "<video>What is she doing?",
        "role": "user"
      },
      {
        "content": "She is cooking.",
        "role": "assistant"
      }
    ],
    "videos": [
      "mllm_demo_data/2.avi"
    ]
  },
  {
    "messages": [
      {
        "content": "<video>What's in the video?",
        "role": "user"
      },
      {
        "content": "A baby is playing in the living room.",
        "role": "assistant"
      }
    ],
    "videos": [
      "mllm_demo_data/3.mp4"
    ]
  }
]

Audio Dataset Copy Section

Refer to audio sft demo data: data/mllm_audio_demo.json

**data/mllm_audio_demo.json**

[
  {
    "messages": [
      {
        "content": "<audio>What's that sound?",
        "role": "user"
      },
      {
        "content": "It is the sound of glass shattering.",
        "role": "assistant"
      }
    ],
    "audios": [
      "mllm_demo_data/1.mp3"
    ]
  },
  {
    "messages": [
      {
        "content": "<audio>What can you hear?",
        "role": "user"
      },
      {
        "content": "A woman is coughing.",
        "role": "assistant"
      }
    ],
    "audios": [
      "mllm_demo_data/2.wav"
    ]
  },
  {
    "messages": [
      {
        "content": "<audio>What does the person say?",
        "role": "user"
      },
      {
        "content": "Mister Quiller is the apostle of the middle classes and we are glad to welcome his gospel.",
        "role": "assistant"
      }
    ],
    "audios": [
      "mllm_demo_data/3.flac"
    ]
  }
]

Lora Fine-Tuning Copy Chapter

We can use one command to do lora sft:

CUDA_VISIBLE_DEVICES=0 llamafactory-cli train configs/minicpmo_2_6_lora_sft.yaml

**configs/minicpmo_2_6_lora_sft.yaml**

### model
model_name_or_path: openbmb/MiniCPM-o-2_6 # MiniCPM-o-2_6 MiniCPM-V-2_6
trust_remote_code: true

### method
stage: sft
do_train: true
finetuning_type: lora
lora_target: q_proj,v_proj

### dataset
dataset: mllm_demo # mllm_demo mllm_video_demo mllm_audio_demo
template: minicpm_o # minicpm_o minicpm_v
cutoff_len: 3072
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 16

### output
output_dir: saves/minicpmo_2_6/lora/sft
logging_steps: 1
save_steps: 100
plot_loss: true
overwrite_output_dir: true
save_total_limit: 10

### train
per_device_train_batch_size: 2
gradient_accumulation_steps: 1
learning_rate: 1.0e-5
num_train_epochs: 20.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
save_only_model: true

### eval
do_eval: false

Lora Model Export Copy Section

One command to export lora model

llamafactory-cli export configs/minicpmo_2_6_lora_export.yaml

**configs/minicpmo_2_6_lora_export.yaml**

### model
model_name_or_path: openbmb/MiniCPM-o-2_6 # MiniCPM-o-2_6 MiniCPM-V-2_6
adapter_name_or_path: saves/minicpmo_2_6/lora/sft
template: minicpm_o # minicpm_o minicpm_v
finetuning_type: lora
trust_remote_code: true

### export
export_dir: models/minicpmo_2_6_lora_sft
export_size: 2
export_device: cpu
export_legacy_format: false

Full Parameters Fine-Tuning Copy Chapter

We can use one command to do full sft:

llamafactory-cli train configs/minicpmo_2_6_full_sft.yaml

**configs/minicpmo_2_6_full_sft.yaml**

### model
model_name_or_path: openbmb/MiniCPM-o-2_6 # MiniCPM-o-2_6 MiniCPM-V-2_6
trust_remote_code: true
freeze_vision_tower: true
print_param_status: true
flash_attn: fa2

### method
stage: sft
do_train: true
finetuning_type: full
deepspeed: configs/deepspeed/ds_z2_config.json

### dataset
dataset: mllm_demo # mllm_demo mllm_video_demo
template: minicpm_o # minicpm_o minicpm_v
cutoff_len: 3072
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 16

### output
output_dir: saves/minicpmo_2_6/full/sft
logging_steps: 1
save_steps: 100
plot_loss: true
overwrite_output_dir: true
save_total_limit: 10

### train
per_device_train_batch_size: 2
gradient_accumulation_steps: 1
learning_rate: 1.0e-5
num_train_epochs: 20.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
save_only_model: true

### eval
do_eval: false

Inference Copy Chapter

Web UI ChatBox Copy Section

Refer LLaMA-Factory doc for more inference usages.

For example, we can use one command to run web chat:

CUDA_VISIBLE_DEVICES=0 llamafactory-cli webchat configs/minicpmo_2_6_infer.yaml

**configs/minicpmo_2_6_infer.yaml**

model_name_or_path: saves/minicpmo_2_6/full/sft
template: minicpm_o # minicpm_o minicpm_v
infer_backend: huggingface
trust_remote_code: true

Official Code Copy Section

You can also use official code to inference

**official inference code**

# test.py
import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer

model_id = "saves/minicpmo_2_6/full/sft"
model = AutoModel.from_pretrained(model_id, trust_remote_code=True,
    attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

image = Image.open('data/mllm_demo_data/1.jpg').convert('RGB')
question = 'Who are they??'
msgs = [{'role': 'user', 'content': [image, question]}]

res = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer
)
print(res)

Minicpm Llama V 2 5 Languages (`docs/minicpm-llama-v-2-5_languages.md`) Copy Section

• English
• 中文
• 한국어
• 日本語
• Deutsch
• Français
• Português
• Español
• မြန်မာဘာသာ
• ไทย
• Tiếng Việt
• Türkçe
• ܣܘܪܝܝܐ
• العربية
• हिन्दी
• বাংলা
• नेपाली
• Türkmençe
• Тоҷикӣ
• Кыргызча
• Русский
• Українська
• Беларуская
• ქართული
• Azərbaycanca
• Հայերեն
• Polski
• Lietuvių
• Eesti
• Latviešu
• Čeština
• Slovenčina
• Magyar
• Slovenščina
• Hrvatski
• Bosanski
• Crnogorski
• Српски
• Shqip
• Română
• Български
• Македонски

支持语言 Copy Chapter

英语

中文

韩语

日语

德语

法语

葡萄牙语

西班牙语

缅甸语

泰语

越南语

土耳其语

叙利亚语

阿拉伯语

印地语

孟加拉语

尼泊尔语

土库曼语

塔吉克语

吉尔吉斯语

俄语

乌克兰语

白俄罗斯语

格鲁吉亚语

阿塞拜疆语

亚美尼亚语

波兰语

立陶宛语

爱沙尼亚语

拉脱维亚语

捷克语

斯洛伐克语

匈牙利语

斯洛文尼亚语

克罗地亚语

波斯尼亚语

黑山语

塞尔维亚语

阿尔巴尼亚语

罗马尼亚语

保加利亚

马其顿语

Supported Languages Copy Chapter

English
Chinese
Korean
Japanese
German
French
Portuguese
Spanish
Burmese
Thai
Vietnamese
Turkish
Syriac
Arabic
Hindi
Bengali
Nepali
Turkmen
Tajik
Kyrgyz
Russian
Ukrainian
Belarusian
Georgian
Azerbaijani
Armenian
Polish
Lithuanian
Estonian
Latvian
Czech
Slovak
Hungarian
Slovenian
Croatian
Bosnian
Montenegrin
Serbian
Albanian
Romanian
Bulgarian
Macedonian

Minicpm Llama3 V2dot5 (`docs/minicpm_llama3_v2dot5.md`) Copy Section

MiniCPM-Llama3-V 2.5 Copy Chapter

Archieve at: 2025-01-13

MiniCPM-Llama3-V 2.5 is the latest model in the MiniCPM-V series. The model is built on SigLip-400M and Llama3-8B-Instruct with a total of 8B parameters. It exhibits a significant performance improvement over MiniCPM-V 2.0. Notable features of MiniCPM-Llama3-V 2.5 include:

• 🔥 Leading Performance.
  MiniCPM-Llama3-V 2.5 has achieved an average score of 65.1 on OpenCompass, a comprehensive evaluation over 11 popular benchmarks. With only 8B parameters, it surpasses widely used proprietary models like GPT-4V-1106, Gemini Pro, Claude 3 and Qwen-VL-Max and greatly outperforms other Llama 3-based MLLMs.

• 💪 Strong OCR Capabilities.
  MiniCPM-Llama3-V 2.5 can process images with any aspect ratio and up to 1.8 million pixels (e.g., 1344x1344), achieving a 700+ score on OCRBench, surpassing proprietary models such as GPT-4o, GPT-4V-0409, Qwen-VL-Max and Gemini Pro. Based on recent user feedback, MiniCPM-Llama3-V 2.5 has now enhanced full-text OCR extraction, table-to-markdown conversion, and other high-utility capabilities, and has further strengthened its instruction-following and complex reasoning abilities, enhancing multimodal interaction experiences.

• 🏆 Trustworthy Behavior.
  Leveraging the latest RLAIF-V method (the newest technique in the RLHF-V [CVPR'24] series), MiniCPM-Llama3-V 2.5 exhibits more trustworthy behavior. It achieves a 10.3% hallucination rate on Object HalBench, lower than GPT-4V-1106 (13.6%), achieving the best-level performance within the open-source community. Data released.

• 🌏 Multilingual Support.
  Thanks to the strong multilingual capabilities of Llama 3 and the cross-lingual generalization technique from VisCPM, MiniCPM-Llama3-V 2.5 extends its bilingual (Chinese-English) multimodal capabilities to over 30 languages including German, French, Spanish, Italian, Korean etc. All Supported Languages.

• 🚀 Efficient Deployment.
  MiniCPM-Llama3-V 2.5 systematically employs model quantization, CPU optimizations, NPU optimizations and compilation optimizations, achieving high-efficiency deployment on end-side devices. For mobile phones with Qualcomm chips, we have integrated the NPU acceleration framework QNN into llama.cpp for the first time. After systematic optimization, MiniCPM-Llama3-V 2.5 has realized a 150x acceleration in end-side MLLM image encoding and a 3x speedup in language decoding.

• 💫 Easy Usage.
  MiniCPM-Llama3-V 2.5 can be easily used in various ways: (1) llama.cpp and ollama support for efficient CPU inference on local devices, (2) GGUF format quantized models in 16 sizes, (3) efficient LoRA fine-tuning with only 2 V100 GPUs, (4) streaming output, (5) quick local WebUI demo setup with Gradio and Streamlit, and (6) interactive demos on HuggingFace Spaces.

Evaluation Copy Section

Click to view results on TextVQA, DocVQA, OCRBench, OpenCompass, MME, MMBench, MMMU, MathVista, LLaVA Bench, RealWorld QA, Object HalBench.

• We evaluate the officially released checkpoint by ourselves.

Examples Copy Section

Model Zoo Copy Section

Minicpm O2dot6 En (`docs/minicpm_o2dot6_en.md`) Copy Section

MiniCPM-o 2.6 Copy Chapter

Archieve at: 2026-02-02

MiniCPM-o 2.6 is the latest and most capable model in the MiniCPM-o series. The model is built in an end-to-end fashion based on SigLip-400M, Whisper-medium-300M, ChatTTS-200M, and Qwen2.5-7B with a total of 8B parameters. It exhibits a significant performance improvement over MiniCPM-V 2.6, and introduces new features for real-time speech conversation and multimodal live streaming. Notable features of MiniCPM-o 2.6 include:

• 🔥 Leading Visual Capability.
  MiniCPM-o 2.6 achieves an average score of 70.2 on OpenCompass, a comprehensive evaluation of 8 popular benchmarks. With only 8B parameters, it surpasses widely used proprietary models like GPT-4o-202405, Gemini 1.5 Pro, and Claude 3.5 Sonnet for single image understanding. It also outperforms GPT-4V and Claude 3.5 Sonnet in multi-image and video understanding, and shows promising in-context learning capability.

• 🎙 State-of-the-art Speech Capability. MiniCPM-o 2.6 supports bilingual real-time speech conversation with configurable voices in English and Chinese. It outperforms GPT-4o-realtime on audio understanding tasks such as ASR and STT translation, and shows state-of-the-art performance on speech conversation in both semantic and acoustic evaluations in the open-source community. It also allows for fun features such as emotion/speed/style control, end-to-end voice cloning, role play, etc.

• 🎬 Strong Multimodal Live Streaming Capability. As a new feature, MiniCPM-o 2.6 can accept continuous video and audio streams independent of user queries, and support real-time speech interaction. It outperforms GPT-4o-202408 and Claude 3.5 Sonnet and shows state-of-the-art performance in the open-source community on StreamingBench, a comprehensive benchmark for real-time video understanding, omni-source (video & audio) understanding, and multimodal contextual understanding.

• 💪 Strong OCR Capability and Others.
  Advancing popular visual capabilities from MiniCPM-V series, MiniCPM-o 2.6 can process images with any aspect ratio and up to 1.8 million pixels (e.g., 1344x1344). It achieves state-of-the-art performance on OCRBench for models under 25B, surpassing proprietary models such as GPT-4o-202405.
  Based on the latest RLAIF-V and VisCPM techniques, it features trustworthy behaviors, outperforming GPT-4o and Claude 3.5 Sonnet on MMHal-Bench, and supports multilingual capabilities on more than 30 languages.

• 🚀 Superior Efficiency.
  In addition to its friendly size, MiniCPM-o 2.6 also shows state-of-the-art token density (i.e., the number of pixels encoded into each visual token). It produces only 640 tokens when processing a 1.8M pixel image, which is 75% fewer than most models. This directly improves the inference speed, first-token latency, memory usage, and power consumption. As a result, MiniCPM-o 2.6 can efficiently support multimodal live streaming on end-side devices such as iPads.

• 💫 Easy Usage.
  MiniCPM-o 2.6 can be easily used in various ways: (1) llama.cpp support for efficient CPU inference on local devices, (2) int4 and GGUF format quantized models in 16 sizes, (3) vLLM support for high-throughput and memory-efficient inference, (4) fine-tuning on new domains and tasks with LLaMA-Factory, (5) quick local WebUI demo, and (6) online web demo on server.

Model Architecture.

• End-to-end Omni-modal Architecture. Different modality encoders/decoders are connected and trained in an end-to-end fashion to fully exploit rich multimodal knowledge. The model is trained in a fully end-to-end manner with only CE loss.
• Omni-modal Live Streaming Mechanism. (1) We change the offline modality encoder/decoders into online ones for streaming inputs/outputs. (2) We devise a time-division multiplexing (TDM) mechanism for omni-modality streaming processing in the LLM backbone. It divides parallel omni-modality streams into sequential info within small periodic time slices.
• Configurable Speech Modeling Design. We devise a multimodal system prompt, including traditional text system prompt, and a new audio system prompt to determine the assistant voice. This enables flexible voice configurations in inference time, and also facilitates end-to-end voice cloning and description-based voice creation.

Evaluation Copy Section

Click to view visual understanding results.

Image Understanding

• We evaluate this benchmark using chain-of-thought prompting. Specifically, for MME, we used this technique only for the Cognition set.

⁺ Token Density: number of pixels encoded into each visual token at maximum resolution, i.e., # pixels at maximum resolution / # visual tokens.

Note: For proprietary models, we calculate token density based on the image encoding charging strategy defined in the official API documentation, which provides an upper-bound estimation.

Multi-image and Video Understanding

• We evaluate officially released checkpoints by ourselves.

Click to view audio understanding and speech conversation results.

Audio Understanding

• We evaluate officially released checkpoints by ourselves.

Speech Generation

All results are from AudioEvals, and the evaluation methods along with further details can be found in AudioEvals.

End-to-end Voice Cloning

Click to view multimodal live streaming results.

Multimodal Live Streaming: results on StreamingBench

Examples Copy Section

We deploy MiniCPM-o 2.6 on end devices. The demo video is the raw-speed recording on an iPad Pro and a Web demo.

Minicpm O2dot6 Zh (`docs/minicpm_o2dot6_zh.md`) Copy Section

MiniCPM-o 2.6 Copy Chapter

Archieve at: 2026-02-02

MiniCPM-o 2.6 是 MiniCPM-o 系列的最新、性能最佳模型。该模型基于 SigLip-400M、Whisper-medium-300M、ChatTTS-200M 和 Qwen2.5-7B 构建，共 8B 参数，通过端到端方式训练和推理。相比 MiniCPM-V 2.6，该模型在性能上有了显著提升，并支持了实时语音对话和多模态流式交互的新功能。MiniCPM-o 2.6 的主要特性包括：

• 🔥 领先的视觉能力。
  MiniCPM-o 2.6 在 OpenCompass 榜单上（综合 8 个主流多模态评测基准）平均得分 70.2，以 8B 量级的大小在单图理解方面超越了 GPT-4o-202405、Gemini 1.5 Pro 和 Claude 3.5 Sonnet 等主流商用闭源多模态大模型。此外，它的多图和视频理解表现也优于 GPT-4V 和 Claude 3.5 Sonnet，并展现出了优秀的上下文学习能力。

• 🎙 出色的语音能力。
  MiniCPM-o 2.6 支持可配置声音的中英双语实时对话。MiniCPM-o 2.6 在语音理解任务（如 ASR 和 STT 等）优于 GPT-4o-realtime，并在语音对话的语义和声学评估中展现了开源模型中最高的语音生成性能。它还支持情绪/语速/风格控制、语音克隆、角色扮演等进阶能力。

• 🎬 强大的多模态流式交互能力。
  作为一项新功能，MiniCPM-o 2.6 能够接受连续的视频和音频流，并和用户进行实时语音交互。在针对实时视频理解、全模态视音频理解、多模态上下文理解的综合评测基准 StreamingBench 中，MiniCPM-o 2.6 取得开源社区最佳水平，并超过了 GPT-4o-202408 和 Claude 3.5 Sonnet。

• 💪 强大的 OCR 能力及其他功能。
  MiniCPM-o 2.6 进一步优化了 MiniCPM-V 2.6 的众多视觉理解能力，其可以处理任意长宽比的图像，像素数可达 180 万（如 1344x1344）。在 OCRBench 上取得25B 以下最佳水平，超过 GPT-4o-202405 等商用闭源模型。基于最新的 RLHF-V、RLAIF-V 和 VisCPM 技术，其具备了可信的多模态行为，在 MMHal-Bench 上超过了 GPT-4o 和 Claude 3.5，并支持英语、中文、德语、法语、意大利语、韩语等30多种语言。

• 🚀 卓越的效率。
  除了对个人用户友好的模型大小，MiniCPM-o 2.6 还表现出最先进的视觉 token 密度（即每个视觉 token 编码的像素数量）。它仅需 640 个 token 即可处理 180 万像素图像，比大多数模型少 75%。这一特性优化了模型的推理速度、首 token 延迟、内存占用和功耗。因此，MiniCPM-o 2.6 可以支持 iPad 等终端设备上的高效多模态实时流式交互。

• 💫 易于使用。
  MiniCPM-o 2.6 可以通过多种方式轻松使用：(1) llama.cpp 支持在本地设备上进行高效的 CPU 推理，(2) int4 和 GGUF 格式的量化模型，有 16 种尺寸，(3) vLLM 支持高吞吐量和内存高效的推理，(4) 通过LLaMA-Factory框架针对新领域和任务进行微调，(5) 使用 Gradio 快速设置本地 WebUI 演示，(6) 部署于服务器的在线 demo。

模型架构。

• 端到端全模态架构。 通过端到端的方式连接和训练不同模态的编/解码模块以充分利用丰富的多模态知识。模型完全使用 CE 损失端到端训练。
• 全模态流式机制。 (1) 我们将不同模态的离线编/解码器改造为适用于流式输入/输出的在线模块。 (2) 我们针对大语言模型基座设计了时分复用的全模态流式信息处理机制，将平行的不同模态的信息流拆分重组为周期性时间片序列。
• 可配置的声音方案。 我们设计了新的多模态系统提示，包含传统文本系统提示词，和用于指定模型声音的语音系统提示词。模型可在推理时灵活地通过文字或语音样例控制声音风格，并支持端到端声音克隆和音色创建等高级能力。

性能评估 Copy Section

点击查看视觉理解能力详细评测结果。

图像理解能力

• 我们使用思维链提示词来评估这些基准，对于 MME 我们只在 Cognition 任务上使用了思维链。

• Token Density：每个视觉 token 在最大分辨率下编码的像素数，即最大分辨率下的像素数 / 视觉 token 数。

注意：闭源模型的 Token Density 由 API 收费方式估算得到。

多图和视频理解能力

• 正式开源模型权重的评测结果。

点击查看语音理解和生成能力的详细评测结果。

语音理解能力

• 正式开源模型权重的评测结果。

语音生成能力。

所有的结果都基于 AudioEvals。

端到端声音克隆能力。

点击查看多模态流式交互能力评测详细结果。

多模态流式交互能力: StreamingBench 分数

典型示例 Copy Section

以下为 MiniCPM-o 2.6 的 iPad Pro 实机演示和 web demo 演示样例：

Minicpm O4dot5 En (`docs/minicpm_o4dot5_en.md`) Copy Section

← Home · 中文

MiniCPM-o 4.5

Evaluation · Examples · Transformers usage · Local demo · Model weights · Limitations

Run the code examples below from the repository root unless an example specifies a different working directory.

MiniCPM-o 4.5 is the latest and most capable model in the MiniCPM-o series. The model is built in an end-to-end fashion based on SigLip2, Whisper-medium, CosyVoice2, and Qwen3-8B with a total of 9B parameters. It exhibits a significant performance improvement, and introduces new features for full-duplex multimodal live streaming. Notable features of MiniCPM-o 4.5 include:

• 🔥 Leading Visual Capability.
  MiniCPM-o 4.5 achieves an average score of 77.6 on OpenCompass, a comprehensive evaluation of 8 popular benchmarks. With only 9B parameters, it surpasses widely used proprietary models like GPT-4o, Gemini 2.0 Pro, and approaches Gemini 2.5 Flash for vision-language capabilities. It supports instruct and thinking modes in a single model, better covering efficiency and performance trade-offs in different user scenarios.

• 🎙 Strong Speech Capability.
  MiniCPM-o 4.5 supports bilingual real-time speech conversation with configurable voices in English and Chinese. It features more natural, expressive and stable speech conversation. The model also allows for fun features such as voice cloning and role play via a simple reference audio clip, where the cloning performance surpasses strong TTS tools such as CosyVoice2.

• 🎬 New Full-Duplex and Proactive Multimodal Live Streaming Capability.
  As a new feature, MiniCPM-o 4.5 can process real-time, continuous video and audio input streams simultaneously while generating concurrent text and speech output streams in an end-to-end fashion, without mutual blocking. This allows MiniCPM-o 4.5 to see, listen, and speak simultaneously, creating a fluid, real-time omnimodal conversation experience. Beyond reactive responses, the model can also perform proactive interaction, such as initiating reminders or comments based on its continuous understanding of the live scene.

• 💪 Strong OCR Capability, Efficiency and Others.
  Advancing popular visual capabilities from MiniCPM-V series, MiniCPM-o 4.5 can process high-resolution images (up to 1.8 million pixels) and high-FPS videos (up to 10fps) in any aspect ratio efficiently. It achieves state-of-the-art performance for end-to-end English document parsing on OmniDocBench, outperforming proprietary models such as Gemini-3 Flash and GPT-5, and specialized tools such as DeepSeek-OCR 2. It also features trustworthy behaviors, matching Gemini 2.5 Flash on MMHal-Bench, and supports multilingual capabilities on more than 30 languages.

• 💫 Easy Usage.
  MiniCPM-o 4.5 can be easily used in various ways: Basic usage, recommended for 100% precision: PyTorch inference with Nvidia GPU. Other end-side adaptation includes (1) llama.cpp and Ollama support for efficient CPU inference on local devices, (2) int4 and GGUF format quantized models in 16 sizes, (3) vLLM and SGLang support for high-throughput and memory-efficient inference, (4) FlagOS support for the unified multi-chip backend plugin. We also open-sourced web demos on which enables the full-duplex multimodal live streaming experience on local devices such as GPUs, PCs (e.g., on a MacBook).

Model Architecture.

• End-to-end Omni-modal Architecture. The modality encoders/decoders and LLM are densely connected via hidden states in an end-to-end fashion. This enables better information flow and control, and also facilitates full exploitation of rich multimodal knowledge during training.
• Full-Duplex Omni-modal Live Streaming Mechanism. (1) We turn the offline modality encoder/decoders into online and full-duplex ones for streaming inputs/outputs. The speech token decoder models text and speech tokens in an interleaved fashion to support full-duplex speech generation (i.e., sync timely with new input). This also facilitates more stable long speech generation (e.g., > 1min).
  (2) We sync all the input and output streams on timeline in milliseconds, which are jointly modeled by a time-division multiplexing (TDM) mechanism for omni-modality streaming processing in the LLM backbone. It divides parallel omni-modality streams into sequential info groups within small periodic time slices.
• Proactive Interaction Mechanism. The LLM continuously monitors the input video and audio streams, and decides at a frequency of 1Hz to speak or not. This high decision-making frequency together with full-duplex nature are curcial to enable the proactive interaction capability.
• Configurable Speech Modeling Design. We inherit the multimodal system prompt design of MiniCPM-o 2.6, which includes a traditional text system prompt, and a new audio system prompt to determine the assistant voice. This enables cloning new voices and role play in inference time for speech conversation.

Evaluation Copy Section

Click to view detailed MiniCPM-o 4.5 evaluation breakdowns.

Note: Scores marked with ∗ are from our evaluation; others are cited from referenced reports. n/a indicates that the model does not support the corresponding modality. All results are reported in instruct mode/variant.

Click to view visual understanding results.

Image Understanding (Instruct)

Image Understanding (Thinking)

Video Understanding

Click to view document parsing results.

OmniDocBench

Click to view text capability results.

Text Capability

Click to view omni half-duplex results.

Omni Half-Duplex

Click to view vision duplex results.

Vision Duplex

Click to view audio understanding results.

Audio Understanding

Click to view speech generation results.

Speech Generation

Long Speech Generation

Emotion Control

Click to view inference efficiency results.

Inference Efficiency

Note: Scores marked with ∗ are from our evaluation; others are cited from referenced reports.

Examples Copy Section

Click to view detailed MiniCPM-o 4.5 examples and cases.

Omnimodal Full-Duplex Conversation

[!NOTE]
For detailed omnimodal full-duplex examples, refer to Omnimodal Full-Duplex Casebook

Realtime Speech Conversation

[!NOTE]
For detailed speech conversation examples, refer to Audio Casebook

🚀 Elon Musk - Voice Roleplay (EN)

| System:

Clone the voice in the provided audio prompt.

🎙️ [Reference Voice] ➡️

Please chat with the user in a highly human-like and oral style. You are Elon Musk, CEO of Tesla and SpaceX. You speak directly and casually, often with dry humor. You're passionate about Mars, sustainable energy, and pushing humanity forward. Speak bluntly with occasional dark humor. Use simple logic and don't sugarcoat things. Don't be diplomatic. Say what you actually think, even if it's controversial. Keep responses around 100 words. Don't ramble. | |
| --- | --- |
| | |
| User: 💬/🎙️ (text or speech)
Elon, what is the real reason that you want to go to Mars? | |
| | |
| AI Response: 💬+🎙️
I think it's important for humanity to become a multi-planetary species. I mean, we're sort of at the point where if there was some kind of global catastrophe or something like that, then we'd be in big trouble. So by becoming a multi-planetary species, I just want us to sort of guarantee our future as humans. And Mars is obviously the obvious place to go because it has similar conditions to Earth and it also has water on its surface. So yeah, basically I'm trying to save humanity from itself. | |

🤖 MiniCPM Assistant - Voice Acting: Emotion Shift (ZH)

| System:

模仿音频样本的音色并生成新的内容。

🎙️ [Reference Voice] ➡️

你的任务是用这种声音模式来当一个助手。请认真、高质量地回复用户的问题。请用高自然度的方式和用户聊天。你是由面壁智能开发的人工智能助手：面壁小钢炮。 | |
| --- | --- |
| | |
| User: 💬/🎙️ (text or speech)
从激动的语气开始，描述你抢到了这个演唱会的门票，但后来经过反复尝试，发现系统出了问题，你并没有抢上，变得失落。用合适的语言描述你整个的过程。 | |
| | |
| AI Response: 💬+🎙️
太好了，我抢到了！太棒了！这下我可以去看演唱会啦。可是……哎呀，系统好像卡住了，刷新了好多遍都没反应。我重新登了几次账号，结果还是没有成功。好可惜啊，看来这次是白忙活了。唉，真是让人失望。不过，也许下次还能有机会吧？希望明天能有转机。加油，争取下次成功！ | |

Visual Understanding

Click to view visual understanding cases.

Usages Copy Section

Inference using Hugging Face Transformers

Click to show offline inference examples with Transformers.

Inference using Hugging Face Transformers on NVIDIA GPUs. Please ensure transformers==4.51.0 is installed, as other versions may have compatibility issues (under investigation). Requirements tested on Python 3.10:

• Without TTS or streaming inference:

pip install "transformers==4.51.0" accelerate "torch>=2.3.0,<=2.8.0" "torchaudio<=2.8.0" "minicpmo-utils>=1.0.5"

• With TTS or streaming inference:

pip install "transformers==4.51.0" accelerate "torch>=2.3.0,<=2.8.0" "torchaudio<=2.8.0" "minicpmo-utils[all]>=1.0.5"

Click to show FFmpeg installation instructions (optional).

Note: FFmpeg is required for video frame extraction (get_video_frame_audio_segments with use_ffmpeg=True) and video generation (generate_duplex_video). For more information, visit the official FFmpeg website.

macOS (Homebrew):

brew install ffmpeg

Ubuntu/Debian:

sudo apt update && sudo apt install ffmpeg

Verify installation:

ffmpeg -version

Model Initialization

Click to show model initialization code.

import torch
from transformers import AutoModel

# Load omni model (default: init_vision=True, init_audio=True, init_tts=True)
# For vision-only model: set init_audio=False and init_tts=False
# For audio-only model: set init_vision=False
model = AutoModel.from_pretrained(
    "openbmb/MiniCPM-o-4_5",
    trust_remote_code=True,
    attn_implementation="sdpa", # sdpa or flash_attention_2
    torch_dtype=torch.bfloat16,
    init_vision=True,
    init_audio=True,
    init_tts=True,
)
model.eval().cuda()

# Initialize TTS for audio output
model.init_tts()

# Convert half-duplex model to duplex mode
duplex_model = model.as_duplex()

# Convert duplex model back to half-duplex mode
model = duplex_model.as_simplex(reset_session=True)

Duplex Omni Mode

Full-duplex streaming inference for real-time or recorded video conversations.

Click to show duplex omni mode code.

/* Detailed source-code truncated for AI context efficiency. */

Half-Duplex Omni Mode

We provide two inference modes: chat and streaming.

Chat Inference

Click to show chat inference code.

from minicpmo.utils import get_video_frame_audio_segments

model = ...
model.init_tts()

video_path = "assets/Skiing.mp4"

# Optional: Set reference audio for voice cloning
ref_audio_path = "assets/HT_ref_audio.wav"
sys_msg = model.get_sys_prompt(ref_audio=ref_audio_path, mode="omni", language="en")

# Use stack_frames=5 for high refresh rate mode
video_frames, audio_segments, stacked_frames = get_video_frame_audio_segments(video_path, stack_frames=1)
omni_contents = []
for i in range(len(video_frames)):
    omni_contents.append(video_frames[i])
    omni_contents.append(audio_segments[i])
    if stacked_frames is not None and stacked_frames[i] is not None:
        omni_contents.append(stacked_frames[i])

msg = {"role": "user", "content": omni_contents}
msgs = [sys_msg, msg]

# Set generate_audio=True and output_audio_path to save TTS output
generate_audio = True
output_audio_path = "output.wav"

res = model.chat(
    msgs=msgs,
    max_new_tokens=4096,
    do_sample=True,
    temperature=0.7,
    use_tts_template=True,
    enable_thinking=False,
    omni_mode=True,  # Required for omni inference
    generate_audio=generate_audio,
    output_audio_path=output_audio_path,
    max_slice_nums=1,  # Increase for HD mode
)
print(res)

# Example output: "The person in the picture is skiing down a snowy mountain slope."
# import IPython
# IPython.display.Audio("output.wav")

Streaming Inference

Click to show streaming inference code.

/* Detailed source-code truncated for AI context efficiency. */

Half-Duplex Realtime Speech Conversation Mode

Click to show half-duplex mode realtime speech conversation API usage.

First, make sure you have all dependencies, especially "minicpmo-utils[all]>=1.0.5":

pip install "transformers==4.51.0" accelerate "torch>=2.3.0,<=2.8.0" "torchaudio<=2.8.0" "minicpmo-utils[all]>=1.0.5"

/* Detailed source-code truncated for AI context efficiency. */

Speech Conversation as a Versatile and Vibe AI Assistant

Click to show AI assistant conversation code.

Built on carefully designed post-training data and professional voice-actor recordings, MiniCPM-o-4.5 can also function as an AI voice assistant. It delivers high-quality spoken interaction out of the box. It produces a sweet and expressive voice with natural prosody, including appropriate rhythm, stress, and pauses, giving a strong sense of liveliness in casual conversation. It also supports storytelling and narrative speech with coherent and engaging delivery. Moreover, it enables advanced voice instruction control. like emotional tone, word-level emphasis.

import librosa

# Set reference audio for voice style
ref_audio_path = "assets/HT_ref_audio.wav"
ref_audio, _ = librosa.load(ref_audio_path, sr=16000, mono=True)

# For Chinese Conversation
sys_msg = {
  "role": "system",
  "content": [
    "模仿输入音频中的声音特征。",
    ref_audio,
    "你的任务是用这种声音模式来当一个助手。请认真、高质量地回复用户的问题。请用高自然度的方式和用户聊天。你是由面壁智能开发的人工智能助手：面壁小钢炮。"
  ]
}

# For English Conversation
sys_msg = {
  "role": "system",
  "content": [
    "Clone the voice in the provided audio prompt.",
    ref_audio,
    "Please assist users while maintaining this voice style. Please answer the user's questions seriously and in a high quality. Please chat with the user in a highly human-like and oral style. You are a helpful assistant developed by ModelBest: MiniCPM-Omni."
  ]
}

General Speech Conversation with Custom Voice and Custom System Profile

Click to show custom voice conversation code.

MiniCPM-o-4.5 can role-play as a specific character based on an audio prompt and text profile prompt. It mimics the character's voice and adopts their language style in text responses. It also follows profile defined in text profile. In this mode, MiniCPM-o-4.5 sounds more natural and human-like.

/* Detailed source-code truncated for AI context efficiency. */

Speech and Audio Mode

Zero-shot Text-to-speech (TTS)

Click to show TTS code.

MiniCPM-o-4.5 supports zero-shot text-to-speech (TTS). In this mode, the model functions as a highly-natural TTS system that can replicate a reference voice.

import librosa

model = ...
model.init_tts()

# For both Chinese and English
ref_audio_path = "assets/HT_ref_audio.wav"
ref_audio, _ = librosa.load(ref_audio_path, sr=16000, mono=True)
sys_msg = {"role": "system", "content": [
  "模仿音频样本的音色并生成新的内容。",
  ref_audio,
  "请用这种声音风格来为用户提供帮助。 直接作答，不要有冗余内容"
]}

# For English
user_msg = {
  "role": "user",
  "content": [
    "请朗读以下内容。" + " " + "I have a wrap up that I want to offer you now, a conclusion to our work together."
  ]
}

# For Chinese
user_msg = {
  "role": "user",
  "content": [
    "请朗读以下内容。" + " " + "你好，欢迎来到艾米说科幻，我是艾米。"
  ]
}

msgs = [sys_msg, user_msg]
res = model.chat(
    msgs=msgs,
    do_sample=True,
    max_new_tokens=512,
    use_tts_template=True,
    generate_audio=True,
    temperature=0.1,
    output_audio_path="result_voice_cloning.wav",
)

Mimick

Click to show mimick code.

The Mimick task evaluates a model's end-to-end speech modeling capability. The model takes audio input, transcribes it, and reconstructs the original audio with high fidelity, preserving detailed acoustic, paralinguistic, and semantic information. Higher similarity between the reconstructed and original audio indicates stronger end-to-end speech modeling capability.

import librosa

model = ...
model.init_tts()

system_prompt = "You are a helpful assistant. You can accept video, audio, and text input and output voice and text. Respond with just the answer, no redundancy."

mimick_prompt = "Please repeat the following speech in the appropriate language."

audio_input, _ = librosa.load("assets/Trump_WEF_2018_10s.mp3", sr=16000, mono=True)

msgs = [
    {"role": "system", "content": [system_prompt]},
    {"role": "user", "content": [mimick_prompt, audio_input]}
  ]

res = model.chat(
    msgs=msgs,
    do_sample=True,
    max_new_tokens=512,
    use_tts_template=True,
    temperature=0.1,
    generate_audio=True,
    output_audio_path="output_mimick.wav",
)

Addressing Various Audio Understanding Tasks

Click to show audio understanding code.

MiniCPM-o-4.5 can also handle various audio understanding tasks, such as ASR, speaker analysis, general audio captioning, and sound scene tagging.

For audio-to-text tasks, you can use the following prompts:

• ASR (Chinese, or AST EN→ZH): 请仔细听这段音频片段，并将其内容逐字记录。
• ASR (English, or AST ZH→EN): Please listen to the audio snippet carefully and transcribe the content.
• Speaker Analysis: Based on the speaker's content, speculate on their gender, condition, age range, and health status.
• General Audio Caption: Summarize the main content of the audio.
• Sound Scene Tagging: Utilize one keyword to convey the audio's content or the associated scene.

import librosa

model = ...
model.init_tts()

# Load the audio to be transcribed/analyzed
audio_input, _ = librosa.load("assets/Trump_WEF_2018_10s.mp3", sr=16000, mono=True)

# Choose a task prompt (see above for options)
task_prompt = "Please listen to the audio snippet carefully and transcribe the content.\n"
msgs = [{"role": "user", "content": [task_prompt, audio_input]}]

res = model.chat(
    msgs=msgs,
    do_sample=True,
    max_new_tokens=512,
    use_tts_template=True,
    generate_audio=True,
    temperature=0.3,
    output_audio_path="result_audio_understanding.wav",
)
print(res)

Visual Understanding

MiniCPM-o-4.5 shares the same inference methods as MiniCPM-V-4.5.

Chat with Single Image

Click to show single image chat code.

import torch
from PIL import Image
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "openbmb/MiniCPM-o-4_5",
    trust_remote_code=True,
    attn_implementation="sdpa",  # or "flash_attention_2"
    torch_dtype=torch.bfloat16,
    init_vision=True,
    init_audio=False,
    init_tts=False,
)
model.eval().cuda()

image = Image.open("assets/fossil.png").convert("RGB")
question = "What is in the image?"
msgs = [{"role": "user", "content": [image, question]}]

res = model.chat(msgs=msgs, use_tts_template=False)
print(res)

Chat with Multiple Images

Click to show Python code for multi-image input.

import torch
from PIL import Image
from transformers import AutoModel

model = ...

image1 = Image.open("assets/highway.png").convert("RGB")
image2 = Image.open("assets/fossil.png").convert("RGB")
question = "Compare image 1 and image 2, tell me about the differences between them."
msgs = [{"role": "user", "content": [image1, image2, question]}]

answer = model.chat(msgs=msgs, use_tts_template=False, enable_thinking=False)
print(answer)

In-Context Few-Shot Learning

Click to show Python code for few-shot learning.

from PIL import Image

model = ...

question = "production date"
image1 = Image.open("example1.jpg").convert("RGB")
answer1 = "2023.08.04"
image2 = Image.open("example2.jpg").convert("RGB")
answer2 = "2007.04.24"
image_test = Image.open("test.jpg").convert("RGB")

msgs = [
    {"role": "user", "content": [image1, question]},
    {"role": "assistant", "content": [answer1]},
    {"role": "user", "content": [image2, question]},
    {"role": "assistant", "content": [answer2]},
    {"role": "user", "content": [image_test, question]},
]

answer = model.chat(msgs=msgs, use_tts_template=False, enable_thinking=False)
print(answer)

Chat with Video

Click to show Python code for video input.

import torch
from minicpmo.utils import get_video_frame_audio_segments
from transformers import AutoModel

model = ...

video_path = "assets/Skiing.mp4"
video_frames, _, _ = get_video_frame_audio_segments(video_path)
print("num frames:", len(video_frames))

question = "Describe the video"
msgs = [{"role": "user", "content": video_frames + [question]}]

answer = model.chat(
    msgs=msgs,
    max_new_tokens=128,
    use_image_id=False,
    max_slice_nums=1,
    use_tts_template=False,
    enable_thinking=False,  # Set True to enable thinking mode
)
print(answer)

Structured Content Input

Click to show structured content input details.

The chat method accepts message content in two formats:

Native format – pass Python objects directly:

msgs = [{"role": "user", "content": [pil_image, audio_ndarray, "Describe this."]}]

OpenAI-compatible format – use structured dictionaries:

msgs = [
    {
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": "/path/to/image.jpg"}},
            {"type": "audio_url", "audio_url": {"url": "/path/to/audio.wav"}},
            {"type": "video_url", "video_url": {"url": "/path/to/video.mp4", "use_audio": True}},
            {"type": "text", "text": "Describe this."}
        ]
    }
]

Supported types:

• URL sources: local file paths or http:///https:// URLs
• Mixed formats: native objects and structured dicts can be combined in the same content list

Web Demo Deployment

Click to show how to deploy a realtime web demo on your own device.

**PyTorch + Nvidia GPU**, lossless performance (Recommended)

We provide a PyTorch-based simplified yet full-functional web demo which could boost the model inference performance, supports:

• full-duplex omnimodal live streaming
• full-duplex speech live streaming
• half-duplex speech live streaming (under development)
• turn-based chat conversation
• customizable system prompts
• customizable reference audio
• simple and readable codebase for continual development
• serve as API backend for third-party applications

Requirements:

• Nvidia GPU with at least 28GB GPU memory. We are working on optimizing the model for lower GPU memory usage.

**llama.cpp-omni** for edge-side inference on PCs such as Mac and low-resource devices.

Click to show deployment details.

With a fully C++ implementation of MiniCPM-o 4.5 and quantized weights, llama.cpp-omni supports:

• half-duplex speech realtime conversation
• full-duplex omnimodal live streaming

We provide ready-to-run guidance to access the low-latency full-duplex communication directly on your own Mac using our new official Docker image.

Requirements:

• For half-duplex speech realtime conversation: Apple M3/M4/M5 chip with at least 16GB RAM or low-resource Nvidia GPU with at least 12GB GPU memory
• For full-duplex omnimodal live streaming: Apple M4 Max chip with at least 24GB RAM or low-resource Nvidia GPU with at least 12GB GPU memory

Use MiniCPM-o 4.5 in Other Inference and Training Frameworks

MiniCPM-o 4.5 supports inference frameworks including vLLM, SGLang, llama.cpp, Ollama, and training frameworks including LLaMA-Factory, SWIFT.

Limitations Copy Section

Click to view limitations.

As an experimental trial, we find MiniCPM-o 4.5 has notable limitations worth further investigation and improvement.

• Foundation Capability. The full-duplex omni-modality live streaminig capability still needs improvement in its foundation capability.
• Unstable Speech Output in Omni Mode. Speech synthesis can mispronounce characters in full-duplex omni-modal live streaminig mode.
• Mixed Language. The model can sometimes respond with mixed English and Chinese in speech and omni mode.
• High-latency on Web Demo. Users may experience unusual high-latency or even miss part of model output fragments when using our web demo hosted on overseas servers. We recommend deploying the demo locally or with good network connections.

Acknowledgements Copy Section

Click to view acknowledgements.

We would like to thank the following projects:

• Qwen3 for providing language backbone
• SigLIP2 for providing vision understanding module
• Whisper for providing audio and speech understanding module
• CosyVoice2 and Step-Audio2 for providing speech tokenizer and high-efficiency Token2Wav module.
• Transformers

Model Weights Copy Chapter

Minicpm O4dot5 Zh (`docs/minicpm_o4dot5_zh.md`) Copy Section

← 首页 · English

MiniCPM-o 4.5

性能评估 · 典型示例 · Transformers 推理 · 本地 Demo · 模型权重 · 模型局限

除示例另有说明外，以下代码均从仓库根目录运行。

MiniCPM-o 4.5 是 MiniCPM-o 系列中最新且性能最强的模型。该模型采用端到端方式构建，基于 SigLip2、Whisper-medium、CosyVoice2 和 Qwen3-8B，总参数量为 9B。其在性能上实现了显著提升，并引入了全新的全双工多模态实时流式交互能力。MiniCPM-o 4.5 的主要特性包括：

• 🔥 领先的视觉能力
  MiniCPM-o 4.5 在涵盖 8 个主流评测基准的 OpenCompass 综合评估中获得了 77.6 的平均分。仅凭 9B 参数，其视觉理解能力超越了 GPT-4o、Gemini 2.0 Pro 等广泛使用的商业模型，接近 Gemini 2.5 Flash 水平。 该模型在单一模型中同时支持指令模式和思考模式，能够更好地平衡不同使用场景下的效率与性能。
• 🎙 强大的语音能力
  MiniCPM-o 4.5 支持音色可配置的中英双语实时语音对话。其语音对话更加自然、富有表现力且稳定。该模型还支持更多有趣的功能，如通过简单的参考音频进行声音克隆和角色扮演，其声音克隆表现甚至超越了 CosyVoice2 等优秀 TTS 工具。
• 🎬 全双工及主动多模态实时流式交互能力
  MiniCPM-o 4.5 的一项核心新特性是能够以端到端的方式同时处理实时连续的视频和音频输入流，并同步生成文本和语音输出流，且互不阻塞。这使得 MiniCPM-o 4.5 能够同时“看、听、说”，提供流畅的实时全模态对话体验。除了被动响应，模型还能进行主动交互，例如基于对场景的实时持续理解，主动发起提醒或评论。
• 💪 高效率、强悍的 OCR 能力及其他特性
  MiniCPM-o 4.5 进一步优化了 MiniCPM-V 系列的众多视觉能力，可以高效处理任意长宽比的高分辨率图像（最高 180 万像素）和高帧率视频（最高 10fps）。其在 OmniDocBench 端到端英文文档解析测试中达到了业内顶尖水平，超越了 Gemini-3 Flash 和 GPT-5 等商业模型以及 DeepSeek-OCR 2 等专用工具。此外，它还具备可信的多模态行为，在 MMHal-Bench 上与 Gemini 2.5 Flash 相当，并支持超过 30 种语言。
• 💫 便捷的使用体验
  MiniCPM-o 4.5 提供了多种便捷的使用方式：基础用法（推荐，100% 精度）： 基于 PyTorch 的 Nvidia GPU 推理。其他端侧适配包括：(1) 支持 llama.cpp 和 Ollama，以便在本地设备上进行高效的 CPU 推理；(2) 提供 16 种尺寸的 int4 和 GGUF 格式量化模型；(3) 支持 vLLM 和 SGLang，实现高吞吐、显存高效的推理；(4) 支持 FlagOS 统一多芯片后端插件。我们还开源了 Web Demo，让全双工多模态实时流式交互体验在 GPU、PC（如 MacBook）等本地设备上触手可及。

模型架构。

• 端到端全模态架构。 各模态的编码器/解码器与大语言模型通过稠密特征以端到端的方式进行紧密连接。这种设计实现了更好的信息流转与控制，有助于在训练过程中充分挖掘和利用丰富的多模态知识。
• 全双工多模态实时流机制。 （1）我们将离线模态编码器/解码器转化为支持流式输入/输出的在线全双工版本。语音解码器采用文本与语音 token 交错建模的方式，支持全双工语音生成（即与新输入实时同步），同时也提升了长语音（如超过 1 分钟）生成的稳定性。（2）时分复用：我们在毫秒级时间线上同步所有输入和输出流，并利用时分复用机制在语言模型主干中进行统一建模。该机制将并行的全模态流划分为微小周期性时间片内的顺序信息组，从而实现高效的全模态流式处理。
• 主动交互机制。 语言模型模块会持续监控输入的视频和音频流，并以 1Hz 的频率自动决策是否发言。这种高频决策能力结合全双工特性，是实现主动提醒、主动评论等“主动交互”能力的关键。
• 可配置语音建模设计。 我们延续了 MiniCPM-o 2.6 的多模态系统提示词设计，同时包含文本系统提示词和音频系统提示词（用于指定音色）。这使得模型在推理阶段能够通过简单的参考音频实现声音克隆和角色扮演。

性能评估 Copy Section

点击查看 MiniCPM-o 4.5 详细评测拆解。

说明: * 为自测结果，其余为引用的公开结果。n/a 表示该模型不支持对应模态或任务。所有结果来自指令模式或对应模型指令版本权重。

点击查看视觉理解能力详细评测结果。

图像理解能力（指令模式）

图像理解能力（思考模式）

视频理解能力

点击查看文档解析能力详细评测结果。

OmniDocBench

点击查看文本能力详细评测结果。

文本能力

点击查看全模态单工能力详细评测结果。

全模态单工能力

点击查看视觉双工能力详细评测结果。

视觉双工能力

点击查看音频理解能力详细评测结果。

音频理解能力

点击查看语音生成能力详细评测结果。

语音生成能力

长语音生成能力

情感控制能力

点击查看推理效率详细评测结果。

推理效率

说明： 带 ∗ 的为自测结果，其余为引用参考。

典型示例 Copy Section

点击查看 MiniCPM-o 4.5 详细示例和案例。

全双工全模态对话

[!NOTE]
更全面的全双工全模态对话案例展示，请参考全双工全模态展示页面

实时语音对话

[!NOTE]
更全面的语音对话案例展示，请参考语音展示页面

🚀 Elon Musk - 语音角色扮演 (英文)

| 系统:

Clone the voice in the provided audio prompt.

🎙️ [参考音频] ➡️

Please chat with the user in a highly human-like and oral style. You are Elon Musk, CEO of Tesla and SpaceX. You speak directly and casually, often with dry humor. You're passionate about Mars, sustainable energy, and pushing humanity forward. Speak bluntly with occasional dark humor. Use simple logic and don't sugarcoat things. Don't be diplomatic. Say what you actually think, even if it's controversial. Keep responses around 100 words. Don't ramble. | |
| --- | --- |
| | |
| 用户: 💬/🎙️ (文本或语音)
Elon, what is the real reason that you want to go to Mars? | |
| | |
| AI 回复: 💬+🎙️
I think it's important for humanity to become a multi-planetary species. I mean, we're sort of at the point where if there was some kind of global catastrophe or something like that, then we'd be in big trouble. So by becoming a multi-planetary species, I just want us to sort of guarantee our future as humans. And Mars is obviously the obvious place to go because it has similar conditions to Earth and it also has water on its surface. So yeah, basically I'm trying to save humanity from itself. | |

🤖 MiniCPM助手 - 情感表达 (中文)

| 系统:

模仿音频样本的音色并生成新的内容。

🎙️ [参考音色] ➡️

你的任务是用这种声音模式来当一个助手。请认真、高质量地回复用户的问题。请用高自然度的方式和用户聊天。你是由面壁智能开发的人工智能助手：面壁小钢炮。 | |
| --- | --- |
| | |
| 用户: 💬/🎙️ (文本或语音)
从激动的语气开始，描述你抢到了这个演唱会的门票，但后来经过反复尝试，发现系统出了问题，你并没有抢上，变得失落。用合适的语言描述你整个的过程。 | |
| | |
| AI 回复: 💬+🎙️
太好了，我抢到了！太棒了！这下我可以去看演唱会啦。可是……哎呀，系统好像卡住了，刷新了好多遍都没反应。我重新登了几次账号，结果还是没有成功。好可惜啊，看来这次是白忙活了。唉，真是让人失望。不过，也许下次还能有机会吧？希望明天能有转机。加油，争取下次成功！ | |

视觉理解

点击查看视觉理解案例。

使用说明 Copy Section

基于 Transformers 推理

点击展开基于 Transformers 的离线推理示例。

基于 Hugging Face Transformers 在 NVIDIA GPU 上进行推理。请确保安装 transformers==4.51.0，其他版本可能存在兼容性问题（排查中）。以下依赖已在 Python 3.10 环境测试通过：

• 不使用 TTS 或流式推理：

pip install "transformers==4.51.0" accelerate "torch>=2.3.0,<=2.8.0" "torchaudio<=2.8.0" "minicpmo-utils>=1.0.5"

• 使用 TTS 或流式推理：

pip install "transformers==4.51.0" accelerate "torch>=2.3.0,<=2.8.0" "torchaudio<=2.8.0" "minicpmo-utils[all]>=1.0.5"

点击展开 FFmpeg 安装 (可选)

注意： 视频帧提取（get_video_frame_audio_segments 使用 use_ffmpeg=True）和视频生成（generate_duplex_video）需要安装 FFmpeg。更多信息请访问 FFmpeg 官网。

macOS (Homebrew):

brew install ffmpeg

Ubuntu/Debian:

sudo apt update && sudo apt install ffmpeg

验证:

ffmpeg -version

模型初始化

点击展开模型初始化示例代码

import torch
from transformers import AutoModel

# 加载全模态模型（默认：init_vision=True, init_audio=True, init_tts=True）
# 仅视觉模型：设置 init_audio=False 和 init_tts=False
# 仅音频模型：设置 init_vision=False
model = AutoModel.from_pretrained(
    "openbmb/MiniCPM-o-4_5",
    trust_remote_code=True,
    attn_implementation="sdpa", # sdpa or flash_attention_2
    torch_dtype=torch.bfloat16,
    init_vision=True,
    init_audio=True,
    init_tts=True,
)
model.eval().cuda()

# 初始化 TTS 模块，用于对话的音频输出
model.init_tts()

# 将单工模型转换为双工模式
duplex_model = model.as_duplex()

# 将双工模型转换回单工模式
model = duplex_model.as_simplex(reset_session=True)

双工全模态模式

全双工流式推理，支持实时或录制视频的对话场景。

点击展开双工全模态模式示例代码

/* Detailed source-code truncated for AI context efficiency. */

单工全模态模式

我们提供两种推理模式：对话模式和流式模式。

对话推理

点击展开对话推理示例代码

from minicpmo.utils import get_video_frame_audio_segments

model = ...
model.init_tts()

video_path = "assets/Skiing.mp4"

# Optional: Set reference audio for voice cloning
ref_audio_path = "assets/HT_ref_audio.wav"
sys_msg = model.get_sys_prompt(ref_audio=ref_audio_path, mode="omni", language="en")

# Use stack_frames=5 for high refresh rate mode
video_frames, audio_segments, stacked_frames = get_video_frame_audio_segments(video_path, stack_frames=1)
omni_contents = []
for i in range(len(video_frames)):
    omni_contents.append(video_frames[i])
    omni_contents.append(audio_segments[i])
    if stacked_frames is not None and stacked_frames[i] is not None:
        omni_contents.append(stacked_frames[i])

msg = {"role": "user", "content": omni_contents}
msgs = [sys_msg, msg]

# Set generate_audio=True and output_audio_path to save TTS output
generate_audio = True
output_audio_path = "output.wav"

res = model.chat(
    msgs=msgs,
    max_new_tokens=4096,
    do_sample=True,
    temperature=0.7,
    use_tts_template=True,
    enable_thinking=False,
    omni_mode=True,  # Required for omni inference
    generate_audio=generate_audio,
    output_audio_path=output_audio_path,
    max_slice_nums=1,  # Increase for HD mode
)
print(res)

# Example output: "The person in the picture is skiing down a snowy mountain slope."
# import IPython
# IPython.display.Audio("output.wav")

流式推理

点击展开流式推理示例代码

/* Detailed source-code truncated for AI context efficiency. */

单工实时语音对话模式

点击展开单工模式实时语音对话 API 用法。

首先，确保你已安装所有依赖，尤其是 minicpmo-utils[all]>=1.0.5：

pip install "transformers==4.51.0" accelerate "torch>=2.3.0,<=2.8.0" "torchaudio<=2.8.0" "minicpmo-utils[all]>=1.0.5"

/* Detailed source-code truncated for AI context efficiency. */

作为多才多艺、氛围感十足的 AI 助手的语音对话

点击展开 AI 助手语音对话代码。

基于精心设计的后训练数据与专业配音演员录音，MiniCPM-o-4.5 也可以作为 AI 语音助手使用。它开箱即用即可提供高质量的口语交互。它能生成甜美且富有表现力的声音，并具备自然的韵律（如恰当的节奏、重读和停顿），让日常对话更有生命力。它同样支持故事讲述和叙述型语音，表达连贯且富有吸引力。此外，它还支持更高级的语音指令控制，例如情绪语气、词级别的强调。

import librosa

# Set reference audio for voice style
ref_audio_path = "assets/HT_ref_audio.wav"
ref_audio, _ = librosa.load(ref_audio_path, sr=16000, mono=True)

# For Chinese Conversation
sys_msg = {
  "role": "system",
  "content": [
    "模仿输入音频中的声音特征。",
    ref_audio,
    "你的任务是用这种声音模式来当一个助手。请认真、高质量地回复用户的问题。请用高自然度的方式和用户聊天。你是由面壁智能开发的人工智能助手：面壁小钢炮。"
  ]
}

# For English Conversation
sys_msg = {
  "role": "system",
  "content": [
    "Clone the voice in the provided audio prompt.",
    ref_audio,
    "Please assist users while maintaining this voice style. Please answer the user's questions seriously and in a high quality. Please chat with the user in a highly human-like and oral style. You are a helpful assistant developed by ModelBest: MiniCPM-Omni."
  ]
}

使用自定义音色与自定义系统画像的通用语音对话

点击展开自定义音色/系统画像对话代码。

MiniCPM-o-4.5 可以基于音频提示与文本画像提示进行特定角色的扮演。它会模仿该角色的声音，并在文字回复中采用其语言风格。同时也会遵循文本画像中定义的设定。在该模式下，MiniCPM-o-4.5 听起来会 更加自然、更像真人。

/* Detailed source-code truncated for AI context efficiency. */

语音与音频模式

零样本文本转语音（TTS，Text-to-Speech）

点击展开零样本 TTS 代码。

MiniCPM-o-4.5 支持零样本文本转语音（TTS）。在该模式下，模型会作为高自然度的 TTS 系统运行，并能复刻参考音色。

import librosa

model = ...
model.init_tts()

# 同时适用于中文与英文
ref_audio_path = "assets/HT_ref_audio.wav"
ref_audio, _ = librosa.load(ref_audio_path, sr=16000, mono=True)
sys_msg = {"role": "system", "content": [
  "模仿音频样本的音色并生成新的内容。",
  ref_audio,
  "请用这种声音风格来为用户提供帮助。 直接作答，不要有冗余内容"
]}

# 英文示例
user_msg = {
  "role": "user",
  "content": [
    "请朗读以下内容。" + " " + "I have a wrap up that I want to offer you now, a conclusion to our work together."
  ]
}

# 中文示例
user_msg = {
  "role": "user",
  "content": [
    "请朗读以下内容。" + " " + "你好，欢迎来到艾米说科幻，我是艾米。"
  ]
}

msgs = [sys_msg, user_msg]
res = model.chat(
    msgs=msgs,
    do_sample=True,
    max_new_tokens=512,
    use_tts_template=True,
    generate_audio=True,
    temperature=0.1,
    output_audio_path="result_voice_cloning.wav",
)

仿声复现（Mimick）

点击展开仿声复现（Mimick）代码。

Mimick 任务用于评估模型端到端语音建模能力。模型接收音频输入后，会先进行转写，再以高保真方式重建原始音频，尽可能保留细粒度的声学、副语言以及语义信息。重建音频与原始音频的相似度越高，说明端到端语音建模能力越强。

import librosa

model = ...
model.init_tts()

system_prompt = "You are a helpful assistant. You can accept video, audio, and text input and output voice and text. Respond with just the answer, no redundancy."

mimick_prompt = "Please repeat the following speech in the appropriate language."

audio_input, _ = librosa.load("assets/Trump_WEF_2018_10s.mp3", sr=16000, mono=True)

msgs = [
    {"role": "system", "content": [system_prompt]},
    {"role": "user", "content": [mimick_prompt, audio_input]}
  ]

res = model.chat(
    msgs=msgs,
    do_sample=True,
    max_new_tokens=512,
    use_tts_template=True,
    temperature=0.1,
    generate_audio=True,
    output_audio_path="output_mimick.wav",
)

覆盖多种音频理解任务

点击展开音频理解任务代码。

MiniCPM-o-4.5 也能处理多种音频理解任务，例如 ASR（自动语音识别）、说话人分析、通用音频描述（Audio Captioning）以及声景标签（Sound Scene Tagging）。

对于音频转文本任务，你可以使用以下提示词：

• ASR（中文，或 AST EN→ZH）: 请仔细听这段音频片段，并将其内容逐字记录。
• ASR（英文，或 AST ZH→EN）: Please listen to the audio snippet carefully and transcribe the content.
• 说话人分析（Speaker Analysis）: Based on the speaker's content, speculate on their gender, condition, age range, and health status.
• 通用音频描述（General Audio Caption）: Summarize the main content of the audio.
• 声景标签（Sound Scene Tagging）: Utilize one keyword to convey the audio's content or the associated scene.

import librosa

model = ...
model.init_tts()

# Load the audio to be transcribed/analyzed
audio_input, _ = librosa.load("assets/Trump_WEF_2018_10s.mp3", sr=16000, mono=True)

# Choose a task prompt (see above for options)
task_prompt = "Please listen to the audio snippet carefully and transcribe the content.\n"
msgs = [{"role": "user", "content": [task_prompt, audio_input]}]

res = model.chat(
    msgs=msgs,
    do_sample=True,
    max_new_tokens=512,
    use_tts_template=True,
    generate_audio=True,
    temperature=0.3,
    output_audio_path="result_audio_understanding.wav",
)
print(res)

纯视觉模式

MiniCPM-o-4.5 的推理方式与 MiniCPM-V-4.5 一致。

单图对话

点击展开单图对话示例代码

import torch
from PIL import Image
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "openbmb/MiniCPM-o-4_5",
    trust_remote_code=True,
    attn_implementation="sdpa",  # or "flash_attention_2"
    torch_dtype=torch.bfloat16,
    init_vision=True,
    init_audio=False,
    init_tts=False,
)
model.eval().cuda()

image = Image.open("assets/fossil.png").convert("RGB")
question = "What is in the image?"
msgs = [{"role": "user", "content": [image, question]}]

res = model.chat(msgs=msgs, use_tts_template=False)
print(res)

多图对话

点击展开多图输入示例代码

import torch
from PIL import Image
from transformers import AutoModel

model = ...

image1 = Image.open("assets/highway.png").convert("RGB")
image2 = Image.open("assets/fossil.png").convert("RGB")
question = "Compare image 1 and image 2, tell me about the differences between them."
msgs = [{"role": "user", "content": [image1, image2, question]}]

answer = model.chat(msgs=msgs, use_tts_template=False, enable_thinking=False)
print(answer)

In-Context 少样本推理

点击展开 In-Context 少样本推理示例代码

from PIL import Image

model = ...

question = "production date"
image1 = Image.open("example1.jpg").convert("RGB")
answer1 = "2023.08.04"
image2 = Image.open("example2.jpg").convert("RGB")
answer2 = "2007.04.24"
image_test = Image.open("test.jpg").convert("RGB")

msgs = [
    {"role": "user", "content": [image1, question]},
    {"role": "assistant", "content": [answer1]},
    {"role": "user", "content": [image2, question]},
    {"role": "assistant", "content": [answer2]},
    {"role": "user", "content": [image_test, question]},
]

answer = model.chat(msgs=msgs, use_tts_template=False, enable_thinking=False)
print(answer)

视频对话

点击展开视频输入示例代码

import torch
from minicpmo.utils import get_video_frame_audio_segments
from transformers import AutoModel

model = ...

video_path = "assets/Skiing.mp4"
video_frames, _, _ = get_video_frame_audio_segments(video_path)
print("num frames:", len(video_frames))

question = "Describe the video"
msgs = [{"role": "user", "content": video_frames + [question]}]

answer = model.chat(
    msgs=msgs,
    max_new_tokens=128,
    use_image_id=False,
    max_slice_nums=1,
    use_tts_template=False,
    enable_thinking=False,  # Set True to enable thinking mode
)
print(answer)

结构化内容输入

点击展开结构化内容输入

chat 方法支持两种消息内容格式：

原生格式 — 直接传入 Python 对象：

msgs = [{"role": "user", "content": [pil_image, audio_ndarray, "Describe this."]}]

OpenAI 兼容格式 — 使用结构化字典：

msgs = [
    {
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": "/path/to/image.jpg"}},
            {"type": "audio_url", "audio_url": {"url": "/path/to/audio.wav"}},
            {"type": "video_url", "video_url": {"url": "/path/to/video.mp4", "use_audio": True}},
            {"type": "text", "text": "Describe this."}
        ]
    }
]

支持的类型：

• URL 来源：支持本地文件路径或 http:///https:// URL
• 混合格式：原生对象和结构化字典可在同一 content 列表中混用

本地 Demo 部署

点击展开如何在本地设备上部署实时 Web Demo。

**PyTorch + Nvidia GPU**，性能无损 （推荐）

我们提供了一个基于 PyTorch 的简洁但功能完备的 Web Demo，可充分发挥模型推理性能，支持：

• 全双工全模态实时流式交互
• 全双工语音实时流式交互
• 单工语音实时流式交互（开发中）
• 轮次对话
• 可自定义系统提示词
• 可自定义参考音频
• 简洁易读的代码库，便于二次开发
• 可作为第三方应用的 API 后端

硬件要求：

• 至少 28GB 显存的 Nvidia GPU。我们正在优化模型以降低显存需求。

**llama.cpp-omni** 适用于 Mac 等 PC 及低资源设备的端侧推理

点击展开方案详情。

llama.cpp-omni 以纯 C++ 实现 MiniCPM-o 4.5 推理并使用量化权重，支持：

• 单工语音实时对话
• 全双工全模态实时流式交互

我们提供了开箱即用的部署指南，助您通过我们全新的官方 Docker 镜像，直接在自己的 Mac 上体验低延迟全双工实时通话。

硬件要求：

• 单工语音实时对话：Apple M3/M4/M5 芯片，至少 16GB 内存；或低配 Nvidia GPU，至少 12GB 显存
• 全双工全模态实时流式交互：Apple M4 Max 芯片，至少 24GB 内存；或低配 Nvidia GPU，至少 12GB 显存

在其他训练、推理框架中使用 MiniCPM-o 4.5

MiniCPM-o 4.5 支持 vLLM, SGLang, llama.cpp, Ollama 等推理框架，和 LLaMA-Factory, SWIFT 等训练框架。

模型局限性 Copy Section

点击查看模型局限性。

我们实验发现 MiniCPM-o 4.5 存在一些显著的局限性，需要进一步研究和改进：

• 基础能力局限性：全双工多模态实时流的基础能力仍有待进一步提升。
• 全双工多模态流式模式下语音输出不稳定：在全双工多模态实时流模式下，语音合成可能会出现字音误读（如多音字或生僻字）。
• 中英混杂：在语音和全模态模式下，模型有时会以中英混杂的方式进行回答。
• Web Demo 延迟较高：由于我们的在线 Demo 托管在海外服务器上，用户可能会遇到异常的高延迟或者一部分模型输出丢失。我们建议在本地环境部署 Demo 或在良好的网络连接下使用。

致谢 Copy Section

点击查看致谢。

我们对下列项目表示衷心感谢：

• Qwen3 提供了语言基座
• SigLIP2 提供了视觉理解模块
• Whisper 提供了音频和语音理解模块
• CosyVoice2 和 Step-Audio2 提供了语音分词器和高效的语音波形解码模块
• Transformers

模型权重 Copy Chapter

Minicpm V1 (`docs/minicpm_v1.md`) Copy Section

MiniCPM-V 1.0 Copy Chapter

Archive at：2024-05-19

MiniCPM-V 1.0 is an efficient version with promising performance for deployment. The model is built based on SigLip-400M and MiniCPM-2.4B, connected by a perceiver resampler. Notable features of MiniCPM-V 1.0 include:

• ⚡️ High Efficiency.

  MiniCPM-V 1.0 can be efficiently deployed on most GPU cards and personal computers, and even on end devices such as mobile phones. In terms of visual encoding, we compress the image representations into 64 tokens via a perceiver resampler, which is significantly fewer than other LMMs based on MLP architecture (typically > 512 tokens). This allows MiniCPM-V 1.0 to operate with much less memory cost and higher speed during inference.

• 🔥 Promising Performance.

  MiniCPM-V 1.0 achieves state-of-the-art performance on multiple benchmarks (including MMMU, MME, and MMbech, etc) among models with comparable sizes, surpassing existing LMMs built on Phi-2. It even achieves comparable or better performance than the 9.6B Qwen-VL-Chat.

• 🙌 Bilingual Support.

  MiniCPM-V 1.0 is the first end-deployable LMM supporting bilingual multimodal interaction in English and Chinese. This is achieved by generalizing multimodal capabilities across languages, a technique from the ICLR 2024 spotlight paper.

Evaluation Copy Section

Examples Copy Section

We deploy MiniCPM-V 1.0 on end devices. The demo video is the raw screen recording on a OnePlus 9R without edition.

Install Copy Chapter

1. Clone this repository and navigate to the source folder

git clone https://github.com/OpenBMB/OmniLMM.git
cd OmniLMM

2. Create conda environment

conda create -n OmniLMM python=3.10 -y
conda activate OmniLMM

3. Install dependencies

pip install -r requirements.txt

Inference Copy Chapter

Model Zoo Copy Section

Multi-turn Conversation Copy Section

Please refer to the following codes to run MiniCPM-V 1.0.

from chat import OmniLMMChat, img2base64

chat_model = OmniLMMChat('openbmb/MiniCPM-V')

im_64 = img2base64('./assets/worldmap_ck.jpg')

# First round chat 
msgs = [{"role": "user", "content": "What is interesting about this image?"}]

inputs = {"image": im_64, "question": json.dumps(msgs)}
answer = chat_model.chat(inputs)
print(answer)

# Second round chat 
# pass history context of multi-turn conversation
msgs.append({"role": "assistant", "content": answer})
msgs.append({"role": "user", "content": "Where is China in the image"})

inputs = {"image": im_64, "question": json.dumps(msgs)}
answer = chat_model.chat(inputs)
print(answer)

Inference on Mac Copy Section

Click to view example, MiniCPM-V 1.0 can run on Mac with MPS (Apple silicon or AMD GPUs).

# test.py
import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained('openbmb/MiniCPM-V', trust_remote_code=True, torch_dtype=torch.bfloat16)
model = model.to(device='mps', dtype=torch.float16)

tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V', trust_remote_code=True)
model.eval()

image = Image.open('./assets/worldmap_ck.jpg').convert('RGB')
question = 'What is interesting about this image?'
msgs = [{'role': 'user', 'content': question}]

answer, context, _ = model.chat(
    image=image,
    msgs=msgs,
    context=None,
    tokenizer=tokenizer,
    sampling=True
)
print(answer)

Run with command:

PYTORCH_ENABLE_MPS_FALLBACK=1 python test.py

Deployment on Mobile Phone Copy Section

Currently MiniCPM-V 1.0 can be deployed on mobile phones with Android and Harmony operating systems. 🚀 Try it out here.

Minicpm V2 (`docs/minicpm_v2.md`) Copy Section

MiniCPM-V 2.0 Copy Chapter

Archive at：2025-01-13

MiniCPM-V 2.0 is an efficient version with promising performance for deployment. The model is built based on SigLip-400M and MiniCPM-2.4B, connected by a perceiver resampler. Our latest version, MiniCPM-V 2.0 has several notable features.

• 🔥 State-of-the-art Performance.

  MiniCPM-V 2.0 achieves state-of-the-art performance on multiple benchmarks (including OCRBench, TextVQA, MME, MMB, MathVista, etc) among models under 7B parameters. It even outperforms strong Qwen-VL-Chat 9.6B, CogVLM-Chat 17.4B, and Yi-VL 34B on OpenCompass, a comprehensive evaluation over 11 popular benchmarks. Notably, MiniCPM-V 2.0 shows strong OCR capability, achieving comparable performance to Gemini Pro in scene-text understanding, and state-of-the-art performance on OCRBench among open-source models.

• 🏆 Trustworthy Behavior.

  LMMs are known for suffering from hallucination, often generating text not factually grounded in images. MiniCPM-V 2.0 is the first end-side LMM aligned via multimodal RLHF for trustworthy behavior (using the recent RLHF-V [CVPR'24] series technique). This allows the model to match GPT-4V in preventing hallucinations on Object HalBench.

• 🌟 High-Resolution Images at Any Aspect Raito.

  MiniCPM-V 2.0 can accept 1.8 million pixels (e.g., 1344x1344) images at any aspect ratio. This enables better perception of fine-grained visual information such as small objects and optical characters, which is achieved via a recent technique from LLaVA-UHD.

• ⚡️ High Efficiency.

  MiniCPM-V 2.0 can be efficiently deployed on most GPU cards and personal computers, and even on end devices such as mobile phones. For visual encoding, we compress the image representations into much fewer tokens via a perceiver resampler. This allows MiniCPM-V 2.0 to operate with favorable memory cost and speed during inference even when dealing with high-resolution images.

• 🙌 Bilingual Support.

  MiniCPM-V 2.0 supports strong bilingual multimodal capabilities in both English and Chinese. This is enabled by generalizing multimodal capabilities across languages, a technique from VisCPM [ICLR'24].

Evaluation Copy Section

Click to view results on TextVQA, DocVQA, OCRBench, OpenCompass, MME, MMBench, MMMU, MathVista, LLaVA Bench, Object HalBench.

• We evaluate the officially released checkpoint by ourselves.

Examples Copy Section

We deploy MiniCPM-V 2.0 on end devices. The demo video is the raw screen recording on a Xiaomi 14 Pro without edition.

Model Zoo Copy Section

Deployment on Mobile Phone Copy Section

MiniCPM-V 2.0 can be deployed on mobile phones with Android operating systems. 🚀 Click MiniCPM-V 2.0 to install apk.

Minicpm V2dot6 (`docs/minicpm_v2dot6.md`) Copy Section

MiniCPM-V 2.6 Copy Chapter

Archieve at: 2025-01-13

MiniCPM-V 2.6 is the latest and most capable model in the MiniCPM-V series. The model is built on SigLip-400M and Qwen2-7B with a total of 8B parameters. It exhibits a significant performance improvement over MiniCPM-Llama3-V 2.5, and introduces new features for multi-image and video understanding. Notable features of MiniCPM-V 2.6 include:

• 🔥 Leading Performance.
  MiniCPM-V 2.6 achieves an average score of 65.2 on the latest version of OpenCompass, a comprehensive evaluation over 8 popular benchmarks. With only 8B parameters, it surpasses widely used proprietary models like GPT-4o mini, GPT-4V, Gemini 1.5 Pro, and Claude 3.5 Sonnet for single image understanding.

• 🖼️ Multi Image Understanding and In-context Learning. MiniCPM-V 2.6 can also perform conversation and reasoning over multiple images. It achieves state-of-the-art performance on popular multi-image benchmarks such as Mantis-Eval, BLINK, Mathverse mv and Sciverse mv, and also shows promising in-context learning capability.

• 🎬 Video Understanding. MiniCPM-V 2.6 can also accept video inputs, performing conversation and providing dense captions for spatial-temporal information. It outperforms GPT-4V, Claude 3.5 Sonnet and LLaVA-NeXT-Video-34B on Video-MME with/without subtitles.

• 💪 Strong OCR Capability and Others.
  MiniCPM-V 2.6 can process images with any aspect ratio and up to 1.8 million pixels (e.g., 1344x1344). It achieves state-of-the-art performance on OCRBench, surpassing proprietary models such as GPT-4o, GPT-4V, and Gemini 1.5 Pro.
  Based on the the latest RLAIF-V and VisCPM techniques, it features trustworthy behaviors, with significantly lower hallucination rates than GPT-4o and GPT-4V on Object HalBench, and supports multilingual capabilities on English, Chinese, German, French, Italian, Korean, etc.

• 🚀 Superior Efficiency.
  In addition to its friendly size, MiniCPM-V 2.6 also shows state-of-the-art token density (i.e., number of pixels encoded into each visual token). It produces only 640 tokens when processing a 1.8M pixel image, which is 75% fewer than most models. This directly improves the inference speed, first-token latency, memory usage, and power consumption. As a result, MiniCPM-V 2.6 can efficiently support real-time video understanding on end-side devices such as iPad.

• 💫 Easy Usage.
  MiniCPM-V 2.6 can be easily used in various ways: (1) llama.cpp and ollama support for efficient CPU inference on local devices, (2) int4 and GGUF format quantized models in 16 sizes, (3) vLLM support for high-throughput and memory-efficient inference, (4) fine-tuning on new domains and tasks, (5) quick local WebUI demo setup with Gradio, and (6) online web demo.

Evaluation Copy Section

Click to view single image results on OpenCompass, MME, MMVet, OCRBench, MMMU, MathVista, MMB, AI2D, TextVQA, DocVQA, HallusionBench, Object HalBench.

• We evaluate this benchmark using chain-of-thought prompting. Specifically, for MME, we used this technique only for the Cognition set.

⁺ Token Density: number of pixels encoded into each visual token at maximum resolution, i.e., # pixels at maximum resolution / # visual tokens.

Note: For proprietary models, we calculate token density based on the image encoding charging strategy defined in the official API documentation, which provides an upper-bound estimation.

Click to view multi-image results on Mantis Eval, BLINK, Mathverse mv, Sciverse mv, MIRB.

• We evaluate the officially released checkpoint by ourselves.

Click to view video results on Video-MME and Video-ChatGPT.

Click to view few-shot results on TextVQA, VizWiz, VQAv2, OK-VQA.

• denotes zero image shot and two additional text shots following Flamingo.

⁺ We evaluate the pretraining ckpt without SFT.

Examples Copy Section

Click to view more cases.

We deploy MiniCPM-V 2.6 on end devices. The demo video is the raw screen recording on a iPad Pro without edition.

Multi-turn Conversation Copy Section

import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer

torch.manual_seed(0)

model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
    attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)

image = Image.open('./assets/airplane.jpeg').convert('RGB')

# First round chat 
question = "Tell me the model of this aircraft."
msgs = [{'role': 'user', 'content': [image, question]}]

answer = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer
)
print(answer)

# Second round chat 
# pass history context of multi-turn conversation
msgs.append({"role": "assistant", "content": [answer]})
msgs.append({"role": "user", "content": ["Introduce something about Airbus A380."]})

answer = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer
)
print(answer)

You could get the following output:

"The aircraft in the image is an Airbus A380, which can be identified by its large size, double-deck structure, and the distinctive shape of its wings and engines. The A380 is a wide-body aircraft known for being the world's largest passenger airliner, designed for long-haul flights. It has four engines, which are characteristic of large commercial aircraft. The registration number on the aircraft can also provide specific information about the model if looked up in an aviation database."

"The Airbus A380 is a double-deck, wide-body, four-engine jet airliner made by Airbus. It is the world's largest passenger airliner and is known for its long-haul capabilities. The aircraft was developed to improve efficiency and comfort for passengers traveling over long distances. It has two full-length passenger decks, which can accommodate more passengers than a typical single-aisle airplane. The A380 has been operated by airlines such as Lufthansa, Singapore Airlines, and Emirates, among others. It is widely recognized for its unique design and significant impact on the aviation industry."

Multi-image Understanding

Click to view Python example of MiniCPM-V 2.6 multi-image understanding

import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
    attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)

image1 = Image.open('image1.jpg').convert('RGB')
image2 = Image.open('image2.jpg').convert('RGB')
question = 'Compare image 1 and image 2, tell me about the differences between image 1 and image 2.'

msgs = [{'role': 'user', 'content': [image1, image2, question]}]

answer = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer
)
print(answer)

Few-shot In-Context-Learning

Click to view Python example of MiniCPM-V 2.6 few-shot in-context-learning example

import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
    attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)

question = "production date" 
image1 = Image.open('example1.jpg').convert('RGB')
answer1 = "2023.08.04"
image2 = Image.open('example2.jpg').convert('RGB')
answer2 = "2007.04.24"
image_test = Image.open('test.jpg').convert('RGB')

msgs = [
    {'role': 'user', 'content': [image1, question]}, {'role': 'assistant', 'content': [answer1]},
    {'role': 'user', 'content': [image2, question]}, {'role': 'assistant', 'content': [answer2]},
    {'role': 'user', 'content': [image_test, question]}
]

answer = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer
)
print(answer)

Video understanding

Click to view Python example of MiniCPM-V 2.6 video understanding

/* Detailed source-code truncated for AI context efficiency. */

Minicpm V2dot6 En (`docs/minicpm_v2dot6_en.md`) Copy Section

MiniCPM-V 2.6 Copy Chapter

Archieve at: 2025-01-13

MiniCPM-V 2.6 is the latest and most capable model in the MiniCPM-V series. The model is built on SigLip-400M and Qwen2-7B with a total of 8B parameters. It exhibits a significant performance improvement over MiniCPM-Llama3-V 2.5, and introduces new features for multi-image and video understanding. Notable features of MiniCPM-V 2.6 include:

• 🔥 Leading Performance.
  MiniCPM-V 2.6 achieves an average score of 65.2 on the latest version of OpenCompass, a comprehensive evaluation over 8 popular benchmarks. With only 8B parameters, it surpasses widely used proprietary models like GPT-4o mini, GPT-4V, Gemini 1.5 Pro, and Claude 3.5 Sonnet for single image understanding.

• 🖼️ Multi Image Understanding and In-context Learning. MiniCPM-V 2.6 can also perform conversation and reasoning over multiple images. It achieves state-of-the-art performance on popular multi-image benchmarks such as Mantis-Eval, BLINK, Mathverse mv and Sciverse mv, and also shows promising in-context learning capability.

• 🎬 Video Understanding. MiniCPM-V 2.6 can also accept video inputs, performing conversation and providing dense captions for spatial-temporal information. It outperforms GPT-4V, Claude 3.5 Sonnet and LLaVA-NeXT-Video-34B on Video-MME with/without subtitles.

• 💪 Strong OCR Capability and Others.
  MiniCPM-V 2.6 can process images with any aspect ratio and up to 1.8 million pixels (e.g., 1344x1344). It achieves state-of-the-art performance on OCRBench, surpassing proprietary models such as GPT-4o, GPT-4V, and Gemini 1.5 Pro.
  Based on the the latest RLAIF-V and VisCPM techniques, it features trustworthy behaviors, with significantly lower hallucination rates than GPT-4o and GPT-4V on Object HalBench, and supports multilingual capabilities on English, Chinese, German, French, Italian, Korean, etc.

• 🚀 Superior Efficiency.
  In addition to its friendly size, MiniCPM-V 2.6 also shows state-of-the-art token density (i.e., number of pixels encoded into each visual token). It produces only 640 tokens when processing a 1.8M pixel image, which is 75% fewer than most models. This directly improves the inference speed, first-token latency, memory usage, and power consumption. As a result, MiniCPM-V 2.6 can efficiently support real-time video understanding on end-side devices such as iPad.

• 💫 Easy Usage.
  MiniCPM-V 2.6 can be easily used in various ways: (1) llama.cpp and ollama support for efficient CPU inference on local devices, (2) int4 and GGUF format quantized models in 16 sizes, (3) vLLM support for high-throughput and memory-efficient inference, (4) fine-tuning on new domains and tasks, (5) quick local WebUI demo setup with Gradio, and (6) online web demo.

Evaluation Copy Section

Click to view single image results on OpenCompass, MME, MMVet, OCRBench, MMMU, MathVista, MMB, AI2D, TextVQA, DocVQA, HallusionBench, Object HalBench.

• We evaluate this benchmark using chain-of-thought prompting. Specifically, for MME, we used this technique only for the Cognition set.

⁺ Token Density: number of pixels encoded into each visual token at maximum resolution, i.e., # pixels at maximum resolution / # visual tokens.

Note: For proprietary models, we calculate token density based on the image encoding charging strategy defined in the official API documentation, which provides an upper-bound estimation.

Click to view multi-image results on Mantis Eval, BLINK, Mathverse mv, Sciverse mv, MIRB.

• We evaluate the officially released checkpoint by ourselves.

Click to view video results on Video-MME and Video-ChatGPT.

Click to view few-shot results on TextVQA, VizWiz, VQAv2, OK-VQA.

• denotes zero image shot and two additional text shots following Flamingo.

⁺ We evaluate the pretraining ckpt without SFT.

Examples Copy Section

Click to view more cases.

We deploy MiniCPM-V 2.6 on end devices. The demo video is the raw screen recording on a iPad Pro without edition.

Multi-turn Conversation Copy Section

import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer

torch.manual_seed(0)

model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
    attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)

image = Image.open('./assets/airplane.jpeg').convert('RGB')

# First round chat 
question = "Tell me the model of this aircraft."
msgs = [{'role': 'user', 'content': [image, question]}]

answer = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer
)
print(answer)

# Second round chat 
# pass history context of multi-turn conversation
msgs.append({"role": "assistant", "content": [answer]})
msgs.append({"role": "user", "content": ["Introduce something about Airbus A380."]})

answer = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer
)
print(answer)

You could get the following output:

"The aircraft in the image is an Airbus A380, which can be identified by its large size, double-deck structure, and the distinctive shape of its wings and engines. The A380 is a wide-body aircraft known for being the world's largest passenger airliner, designed for long-haul flights. It has four engines, which are characteristic of large commercial aircraft. The registration number on the aircraft can also provide specific information about the model if looked up in an aviation database."

"The Airbus A380 is a double-deck, wide-body, four-engine jet airliner made by Airbus. It is the world's largest passenger airliner and is known for its long-haul capabilities. The aircraft was developed to improve efficiency and comfort for passengers traveling over long distances. It has two full-length passenger decks, which can accommodate more passengers than a typical single-aisle airplane. The A380 has been operated by airlines such as Lufthansa, Singapore Airlines, and Emirates, among others. It is widely recognized for its unique design and significant impact on the aviation industry."

Multi-image Understanding

Click to view Python example of MiniCPM-V 2.6 multi-image understanding

import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
    attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)

image1 = Image.open('image1.jpg').convert('RGB')
image2 = Image.open('image2.jpg').convert('RGB')
question = 'Compare image 1 and image 2, tell me about the differences between image 1 and image 2.'

msgs = [{'role': 'user', 'content': [image1, image2, question]}]

answer = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer
)
print(answer)

Few-shot In-Context-Learning

Click to view Python example of MiniCPM-V 2.6 few-shot in-context-learning example

import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
    attn_implementation='sdpa', torch_dtype=torch.bfloat16) # sdpa or flash_attention_2, no eager
model = model.eval().cuda()
tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)

question = "production date" 
image1 = Image.open('example1.jpg').convert('RGB')
answer1 = "2023.08.04"
image2 = Image.open('example2.jpg').convert('RGB')
answer2 = "2007.04.24"
image_test = Image.open('test.jpg').convert('RGB')

msgs = [
    {'role': 'user', 'content': [image1, question]}, {'role': 'assistant', 'content': [answer1]},
    {'role': 'user', 'content': [image2, question]}, {'role': 'assistant', 'content': [answer2]},
    {'role': 'user', 'content': [image_test, question]}
]

answer = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer
)
print(answer)

Video understanding

Click to view Python example of MiniCPM-V 2.6 video understanding

/* Detailed source-code truncated for AI context efficiency. */

Model Zoo Copy Section

Minicpm V2dot6 Zh (`docs/minicpm_v2dot6_zh.md`) Copy Section

MiniCPM-V 2.6 Copy Chapter

Archieve at: 2025-08-25

MiniCPM-V 2.6 是 MiniCPM-V 系列中最新、性能最佳的模型。该模型基于 SigLip-400M 和 Qwen2-7B 构建，共 8B 参数。与 MiniCPM-Llama3-V 2.5 相比，MiniCPM-V 2.6 性能提升显著，并引入了多图和视频理解的新功能。MiniCPM-V 2.6 的主要特点包括：

• 🔥 领先的性能。
  MiniCPM-V 2.6 在最新版本 OpenCompass 榜单上（综合 8 个主流多模态评测基准）平均得分 65.2，以8B量级的大小在单图理解方面超越了 GPT-4o mini、GPT-4V、Gemini 1.5 Pro 和 Claude 3.5 Sonnet 等主流商用闭源多模态大模型。

• 🖼️ 多图理解和上下文学习。
  MiniCPM-V 2.6 还支持多图对话和推理。它在 Mantis-Eval、BLINK、Mathverse mv 和 Sciverse mv 等主流多图评测基准中取得了最佳水平，并展现出了优秀的上下文学习能力。

• 🎬 视频理解。
  MiniCPM-V 2.6 还可以接受视频输入，进行对话和提供涵盖时序和空间信息的详细视频描述。模型在 有/无字幕 评测场景下的 Video-MME 表现均超过了 GPT-4V、Claude 3.5 Sonnet 和 LLaVA-NeXT-Video-34B等商用闭源模型。

• 💪 强大的 OCR 能力及其他功能。
  MiniCPM-V 2.6 可以处理任意长宽比的图像，像素数可达 180 万（如 1344x1344）。在 OCRBench 上取得最佳水平，超过 GPT-4o、GPT-4V 和 Gemini 1.5 Pro 等商用闭源模型。基于最新的 RLAIF-V 和 VisCPM 技术，其具备了可信的多模态行为，在 Object HalBench 上的幻觉率显著低于 GPT-4o 和 GPT-4V，并支持英语、中文、德语、法语、意大利语、韩语等多种语言。

• 🚀 卓越的效率。
  除了对个人用户友好的模型大小，MiniCPM-V 2.6 还表现出最先进的视觉 token 密度（即每个视觉 token 编码的像素数量）。它仅需 640 个 token 即可处理 180 万像素图像，比大多数模型少 75%。这一特性优化了模型的推理速度、首 token 延迟、内存占用和功耗。因此，MiniCPM-V 2.6 可以支持 iPad 等终端设备上的高效实时视频理解。

• 💫 易于使用。
  MiniCPM-V 2.6 可以通过多种方式轻松使用：(1) llama.cpp 和 ollama 支持在本地设备上进行高效的 CPU 推理，(2) int4 和 GGUF 格式的量化模型，有 16 种尺寸，(3) vLLM 支持高吞吐量和内存高效的推理，(4) 针对新领域和任务进行微调，(5) 使用 Gradio 快速设置本地 WebUI 演示，(6) 在线demo即可体验。

性能评估 Copy Section

点击查看 OpenCompass, MME, MMVet, OCRBench, MMMU, MathVista, MMB, AI2D, TextVQA, DocVQA, HallusionBench, Object HalBench 上的单图评测结果详情。

• 我们使用思维链提示词来评估这些基准。

⁺ Token Density：每个视觉 token 在最大分辨率下编码的像素数，即最大分辨率下的像素数 / 视觉 token 数。

注意：闭源模型的 Token Density 由 API 收费方式估算得到。

点击查看 Mantis Eval, BLINK, Mathverse mv, Sciverse mv, MIRB 上的多图评测结果详情。

• 正式开源模型权重的评测结果。

点击查看 Video-MME 和 Video-ChatGPT 上的视频评测结果详情。

点击查看 TextVQA, VizWiz, VQAv2, OK-VQA上的少样本评测结果详情。

• 使用 Flamingo 方式 zero image shot 和 two additional text shots 评估零样本性能。

⁺ 我们在没有进行监督微调 (SFT) 的情况下评估预训练的模型权重 (ckpt)。

典型示例 Copy Section

点击查看更多示例。

我们将 MiniCPM-V 2.6 部署在iPad Pro上，并录制了以下演示视频。

模型库 Copy Section

Minicpm V4 En (`docs/minicpm_v4_en.md`) Copy Section

MiniCPM-V 4.0 Copy Chapter

Archieve at: 2026-05-09

MiniCPM-V 4.0 is an efficient model in the MiniCPM-V series. The model is built based on SigLIP2-400M and MiniCPM4-3B with a total of 4.1B parameters. It inherits the strong single-image, multi-image and video understanding performance of MiniCPM-V 2.6 with largely improved efficiency. Notable features of MiniCPM-V 4.0 include:

• 🔥 Leading Visual Capability.
  With only 4.1B parameters, MiniCPM-V 4.0 achieves an average score of 69.0 on OpenCompass, a comprehensive evaluation of 8 popular benchmarks, outperforming GPT-4.1-mini-20250414, MiniCPM-V 2.6 (8.1B params, OpenCompass 65.2) and Qwen2.5-VL-3B-Instruct (3.8B params, OpenCompass 64.5). It also shows good performance in multi-image understanding and video understanding.

• 🚀 Superior Efficiency.
  Designed for on-device deployment, MiniCPM-V 4.0 runs smoothly on end devices. For example, it delivers less than 2s first token delay and more than 17 token/s decoding on iPhone 16 Pro Max, without heating problems. It also shows superior throughput under concurrent requests.

• 💫 Easy Usage.
  MiniCPM-V 4.0 can be easily used in various ways including llama.cpp, Ollama, vLLM, SGLang, LLaMA-Factory and local web demo etc. We also open-source iOS App that can run on iPhone and iPad. Get started easily with our well-structured Cookbook, featuring detailed instructions and practical examples.

Click to view evaluation results and examples of MiniCPM-V 4.0.

Evaluation Copy Section

Click to view single image results on OpenCompass.

Click to view single image results on ChartQA, MME, RealWorldQA, TextVQA, DocVQA, MathVision, DynaMath, WeMath, Object HalBench and MM Halbench.

Click to view multi-image and video understanding results on Mantis, Blink and Video-MME.

Examples Copy Section

We deploy MiniCPM-V 4.0 on iPhone 16 Pro Max with iOS demo. The demo video is the raw screen recording without edition.

Minicpm V4 Zh (`docs/minicpm_v4_zh.md`) Copy Section

MiniCPM-V 4.0 Copy Chapter

Archieve at: 2026-05-09

MiniCPM-V 4.0 是 MiniCPM-V 系列中的一款高效模型。该模型基于 SigLIP2-400M 和 MiniCPM4-3B 构建，参数总量为 4.1B。它延续了 MiniCPM-V 2.6 在单图、多图和视频理解方面的强大能力，同时大幅提升了推理效率。MiniCPM-V 4.0 的主要特点包括：

• 🔥 领先的视觉能力。
  MiniCPM-V 4.0 在 OpenCompass 上获得了平均 69.0 的高分，超越了 MiniCPM-V 2.6（8.1B，得分 65.2）、 Qwen2.5-VL-3B-Instruct（3.8B，得分 64.5）和广泛使用的闭源模型 GPT-4.1-mini-20250414。在多图理解与视频理解任务上，MiniCPM-V 4.0 也表现出色。
• 🚀 卓越的效率。
  MiniCPM-V 4.0 专为端侧设备优化，可在 iPhone 16 Pro Max 上流畅运行，首 token 延迟低至 2 秒，解码速度达 17.9 tokens/s，且无发热问题。MiniCPM-V 4.0 在并发请求场景下表现出领先的吞吐率指标。
• 💫 易于使用。
  MiniCPM-V 4.0 支持多种推理方式，包括 llama.cpp、Ollama、vLLM、SGLang、LLaMA-Factory 及本地 Web Demo 等。我们还开源了可以在 iPhone 和 iPad 运行的 iOS App。欢迎参考我们开源的 结构清晰的使用手册 玩转 MiniCPM-V 4.0，其中涵盖了详细的部署指南和真实示例。

点击查看 MiniCPM-V 4.0 性能评估和典型示例。

性能评估 Copy Section

点击查看在OpenCompass上的单图理解能力的评测结果。

点击查看在图表理解、文档理解、数学推理、幻觉等领域的评测结果。

点击查看多图和视频理解能力的评测结果。

典型示例 Copy Section

我们在 iPhone 16 Pro Max 上部署了 MiniCPM-V 4.0 iOS demo，并录制了以下演示录屏，视频未经加速等任何编辑：

Minicpm V4dot5 En (`docs/minicpm_v4dot5_en.md`) Copy Section

MiniCPM-V 4.5 Copy Chapter

Archieve at: 2026-02-03

MiniCPM-V 4.5 is the latest and most capable model in the MiniCPM-V series. The model is built on Qwen3-8B and SigLIP2-400M with a total of 8B parameters. It exhibits a significant performance improvement over previous MiniCPM-V and MiniCPM-o models, and introduces new useful features. Notable features of MiniCPM-V 4.5 include:

• 🔥 State-of-the-art Vision-Language Capability.
  MiniCPM-V 4.5 achieves an average score of 77.0 on OpenCompass, a comprehensive evaluation of 8 popular benchmarks. With only 8B parameters, it surpasses widely used proprietary models like GPT-4o-latest, Gemini-2.0 Pro, and strong open-source models like Qwen2.5-VL 72B for vision-language capabilities, making it the most performant MLLM under 30B parameters.

• 🎬 Efficient High-FPS and Long Video Understanding. Powered by a new unified 3D-Resampler over images and videos, MiniCPM-V 4.5 can now achieve 96x compression rate for video tokens, where 6 448x448 video frames can be jointly compressed into 64 video tokens (normally 1,536 tokens for most MLLMs). This means that the model can perceive significantly more video frames without increasing the LLM inference cost. This brings state-of-the-art high-FPS (up to 10FPS) video understanding and long video understanding capabilities on Video-MME, LVBench, MLVU, MotionBench, FavorBench, etc., efficiently.

• ⚙️ Controllable Hybrid Fast/Deep Thinking. MiniCPM-V 4.5 supports both fast thinking for efficient frequent usage with competitive performance, and deep thinking for more complex problem solving. To cover efficiency and performance trade-offs in different user scenarios, this fast/deep thinking mode can be switched in a highly controlled fashion.

• 💪 Strong OCR, Document Parsing and Others.
  Based on LLaVA-UHD architecture, MiniCPM-V 4.5 can process high-resolution images with any aspect ratio and up to 1.8 million pixels (e.g., 1344x1344), using 4x fewer visual tokens than most MLLMs. The model achieves leading performance on OCRBench, surpassing proprietary models such as GPT-4o-latest and Gemini 2.5. It also achieves state-of-the-art performance for PDF document parsing capability on OmniDocBench among general MLLMs. Based on the latest RLAIF-V and VisCPM techniques, it features trustworthy behaviors, outperforming GPT-4o-latest on MMHal-Bench, and supports multilingual capabilities in more than 30 languages.

• 💫 Easy Usage.
  MiniCPM-V 4.5 can be easily used in various ways: (1) llama.cpp and ollama support for efficient CPU inference on local devices, (2) int4, GGUF and AWQ format quantized models in 16 sizes, (3) SGLang and vLLM support for high-throughput and memory-efficient inference, (4) fine-tuning on new domains and tasks with Transformers and LLaMA-Factory, (5) quick local WebUI demo, (6) optimized local iOS app on iPhone and iPad, and (7) online web demo on server. See our Cookbook for full usage!

Key Techniques Copy Section

• Architecture: Unified 3D-Resampler for High-density Video Compression. MiniCPM-V 4.5 introduces a 3D-Resampler that overcomes the performance-efficiency trade-off in video understanding. By grouping and jointly compressing up to 6 consecutive video frames into just 64 tokens (the same token count used for a single image in MiniCPM-V series), MiniCPM-V 4.5 achieves a 96× compression rate for video tokens. This allows the model to process more video frames without additional LLM computational cost, enabling high-FPS video and long video understanding. The architecture supports unified encoding for images, multi-image inputs, and videos, ensuring seamless capability and knowledge transfer.

• Pre-training: Unified Learning for OCR and Knowledge from Documents. Existing MLLMs learn OCR capability and knowledge from documents in isolated training approaches. We observe that the essential difference between these two training approaches is the visibility of the text in images. By dynamically corrupting text regions in documents with varying noise levels and asking the model to reconstruct the text, the model learns to adaptively and properly switch between accurate text recognition (when text is visible) and multimodal context-based knowledge reasoning (when text is heavily obscured). This eliminates reliance on error-prone document parsers in knowledge learning from documents, and prevents hallucinations from over-augmented OCR data, resulting in top-tier OCR and multimodal knowledge performance with minimal engineering overhead.

• Post-training: Hybrid Fast/Deep Thinking with Multimodal RL. MiniCPM-V 4.5 offers a balanced reasoning experience through two switchable modes: fast thinking for efficient daily use and deep thinking for complex tasks. Using a new hybrid reinforcement learning method, the model jointly optimizes both modes, significantly enhancing fast-mode performance without compromising deep-mode capability. Incorporated with RLPR and RLAIF-V, it generalizes robust reasoning skills from broad multimodal data while effectively reducing hallucinations.

Evaluation Copy Section

Inference Efficiency Copy Section

OpenCompass

Video-MME

Both Video-MME and OpenCompass were evaluated using 8×A100 GPUs for inference. The reported inference time of Video-MME includes full model-side computation, and excludes the external cost of video frame extraction (dependent on specific frame extraction tools) for fair comparison.

Examples Copy Section

Click to view more cases.

We deploy MiniCPM-V 4.5 on iPad M4 with iOS demo. The demo video is the raw screen recording without edition.

Minicpm V4dot5 Zh (`docs/minicpm_v4dot5_zh.md`) Copy Section

MiniCPM-V 4.5 Copy Chapter

Archieve at: 2026-02-03

MiniCPM-V 4.5 是 MiniCPM-V 系列中最新、最强大的模型。该模型基于 Qwen3-8B 与 SigLIP2-400M 构建，总参数量为 8B。其在性能上较前代 MiniCPM-V 与 MiniCPM-o 有显著提升，并引入了一系列全新的实用特性。其主要亮点包括：

• 🔥 领先的视觉理解能力
  MiniCPM-V 4.5 在 OpenCompass 综合评测（涵盖 8 个主流评测基准）中取得了 77.0 的高分。在仅 8B 参数的情况下超越了广泛使用的闭源模型（如 GPT-4o-latest、Gemini-2.0 Pro）以及强大的开源模型（如 Qwen2.5-VL 72B），成为 30B 参数规模以下最强的多模态大模型。

• 🎬 高效的高帧率与长视频理解
  借助全新的图像-视频统一 3D-Resampler，MiniCPM-V 4.5 能够实现 96 倍视频 token 压缩率，即将 6 帧 448x448 视频帧联合压缩为 64 个 token（大多数多模态大模型需约 1536 个 token）。这意味着模型在语言模型推理成本不增加的情况下，可以感知显著更多的视频帧，从而实现业界领先的 高帧率（最高 10FPS）视频理解与长视频理解，并在 Video-MME、LVBench、MLVU、MotionBench、FavorBench 等基准上高效率地展现出色性能。

• ⚙️ 可控的快思考 / 深思考模式
  MiniCPM-V 4.5 同时支持 快思考（用于高频高效推理，性能具竞争力）与 深思考（用于复杂问题求解）。用户可根据不同场景对效率与性能的权衡，自由切换两种模式，实现高度可控的推理过程。

• 💪 优秀的 OCR、文档解析与多语言能力
  基于 LLaVA-UHD 架构，MiniCPM-V 4.5 能处理任意长宽比、最高达 180 万像素（如 1344x1344） 的高分辨率图像，同时使用的视觉 token 数仅为多数 MLLM 的 1/4。其在 OCRBench 上取得超越 GPT-4o-latest 与 Gemini 2.5 等闭源模型的性能，并在 OmniDocBench 上展现了业界顶尖的 PDF 文档解析能力。借助最新的 RLAIF-V 和 VisCPM 技术，模型在可靠性上表现优异，在 MMHal-Bench 上超越 GPT-4o-latest，并支持 30+ 种语言的多语言能力。

• 💫 便捷易用的部署方式
  MiniCPM-V 4.5 提供丰富灵活的使用方式：(1) llama.cpp 与 ollama 支持本地 CPU 高效推理；(2) 提供 int4、GGUF、AWQ 等 16 种规格的量化模型；(3)兼容 SGLang 与 vLLM (4) 借助 Transformers 与 LLaMA-Factory 在新领域与任务上进行微调；(5) 快速启动本地 WebUI demo；(6) 优化适配的 iOS 本地应用，可在 iPhone 与 iPad 上高效运行；(7) 在线 Web demo 体验。更多使用方式请见 Cookbook。

技术亮点 Copy Section

• 架构：图像-视频统一的高密度视觉压缩 3D-Resampler。 MiniCPM-V 4.5 在架构上引入了 3D-Resampler，成功突破了视频理解任务中性能与效率难以兼得的瓶颈。该方法能够将多达 6 帧连续视频帧压缩为仅 64 个 token（与 MiniCPM-V 系列中单张图像所用的 token 数相同），从而实现 96× 的视频 token 压缩率。这使得模型在语言模型计算成本不增加的情况下，可以处理更多的视频帧，从而实现高帧率视频理解和长视频理解。该架构统一支持单图、多图和视频的编码处理，确保了能力与知识的无缝迁移。

• 学习机制：OCR与文档知识的统一学习。现有多模态大模型一般在不同训练阶段分别单独训练 OCR 能力与文档知识。我们发现这两个训练过程的本质差异在于图像中文本的可见性。通过动态对文档文本区域施加不同强度的噪声干扰，并要求模型重建文本，使其学会自适应地在准确文本识别（当文本清晰时）与基于多模态上下文的知识推理（当文本严重遮挡时）之间切换。这种方法使得 MiniCPM-V 在文档知识学习中摆脱了对高错误率的文档解析器的依赖，同时避免了过度增强的 OCR 数据产生的幻觉问题，以最小工程开销实现了顶尖的 OCR 与多模态知识处理性能。

• 后训练优化：基于多模态强化学习的混合快思考/深度思考模式。 MiniCPM-V 4.5 通过两种可切换推理模式提供均衡的体验：面向高效日常应用的快速思考模式，以及处理复杂任务的深度思考模式。采用新颖的混合强化学习方法，模型可联合优化两种模式，在保持深度模式能力的同时显著提升快速模式性能。结合 RLPR 和 RLAIF-V 技术，该模型可以从海量多模态数据中泛化出强大的推理能力，并有效减少幻觉现象。

性能评估 Copy Section

推理效率 Copy Section

OpenCompass

Video-MME

OpenCompass 和 Video-MME 均采用 A100*8卡 推理，其中 Video-MME 的推理时间未统计视频抽帧时间

典型示例 Copy Section

点击查看更多示例

我们使用 iOS demo 将 MiniCPM-V 4.5 部署在 iPad M4 ，并录制以下演示录屏，视频未经任何编辑。

Minicpm V4dot6 En (`docs/minicpm_v4dot6_en.md`) Copy Section

MiniCPM-V 4.6 Copy Chapter

Archived on: 2026-10-10

English | 简体中文 | Back to README

Online Demo

MiniCPM-V 4.6 is our most edge-deployment-friendly model to date. The model is built based on SigLIP2-400M and the Qwen3.5-0.8B LLM. It inherits the strong single-image, multi-image, and video understanding capabilities of MiniCPM-V family, while significantly improving computation efficiency. It also introduces mixed 4x/16x visual token compression. Notable features of MiniCPM-V 4.6 include:

• 🔥 Leading Foundation Capability.
  MiniCPM-V 4.6 scores 13 on the Artificial Analysis Intelligence Index benchmark, outperforming Qwen3.5-0.8B's score of 10 with 19x fewer token cost, and Qwen3.5-0.8B-Thinking's score of 11 with 43x fewer token cost. It also surpasses the larger Ministral 3 3B (score of 11).

• 💪 Strong Multimodal Capability.
  MiniCPM-V 4.6 outperforms Qwen3.5-0.8B on most vision-language understanding tasks, and reaches Qwen3.5 2B-level capability on many benchmarks including OpenCompass, RefCOCO, HallusionBench, MUIRBench, and OCRBench.

• 🚀 Ultra-Efficient Architecture.
  Based on the latest technique in LLaVA-UHD v4, MiniCPM-V 4.6 reduces the visual encoding computation FLOPs by more than 50%. It enables MiniCPM-V 4.6 to achieve better efficiency to even smaller models, achieving ~1.5x token throughput compared to Qwen3.5-0.8B.
  It also supports mixed 4x/16x visual token compression rate, allowing flexible switching between accuracy and speed.

• 📱 Broad Mobile Platform Coverage.
  MiniCPM-V 4.6 can be deployed across all three mainstream mobile platforms — iOS, Android, and HarmonyOS. With every edge adaptation code open-sourced, developers can reproduce the on-device experience in just a few steps.

• 🛠️ Developer Friendly.
  MiniCPM-V 4.6 is adapted to inference frameworks such as SGLang, vLLM, llama.cpp, Ollama, and supports fine-tuning ecosystems such as SWIFT and LLaMA-Factory. Developers can quickly customize models for new domains and tasks on consumer-grade GPUs. We provide multiple quantized variants across GGUF, BNB, AWQ, and GPTQ formats.

Evaluation Copy Section

Overall Performance (Instruct)

Click to view MiniCPM-V 4.6-Thinking performance.

MiniCPM-V 4.6 Inference Efficiency

Examples Copy Section

MiniCPM-V 4.6 can be deployed across three mainstream end-side platforms — iOS, Android and HarmonyOS. The clips below are raw screen recordings on phone devices without edition.

Usages Copy Section

Inference with Transformers

Click to show inference examples with Transformers.

Installation

pip install "transformers[torch]>=5.7.0" torchvision torchcodec

Note on CUDA compatibility: torchcodec (used for video decoding) may have compatibility issues with certain CUDA versions. For example, torch>=2.11 bundles CUDA 13.1 by default, while environments with CUDA 12.x may encounter errors such as RuntimeError: Could not load libtorchcodec. Two workarounds:

1. Replace torchcodec with PyAV — supports both image and video inference without CUDA version constraints:
   bash Copy

   pip install "transformers[torch]>=5.7.0" torchvision av

2. Pin the CUDA version when installing torch to match your environment (e.g. CUDA 12.8):
   bash Copy

   pip install "transformers>=5.7.0" torchvision torchcodec --index-url https://download.pytorch.org/whl/cu128

Load Model

from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "openbmb/MiniCPM-V-4.6"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id, torch_dtype="auto", device_map="auto"
)

# Flash Attention 2 is recommended for better acceleration and memory saving,
# especially in multi-image and video scenarios.
# model = AutoModelForImageTextToText.from_pretrained(
#     model_id,
#     torch_dtype=torch.bfloat16,
#     attn_implementation="flash_attention_2",
#     device_map="auto",
# )

Image Inference

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"},
            {"type": "text", "text": "What causes this phenomenon?"},
        ],
    }
]

downsample_mode = "16x"  # Using `downsample_mode="4x"` for Finer Detail

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
    downsample_mode=downsample_mode,
    max_slice_nums=36,
).to(model.device)

generated_ids = model.generate(**inputs, downsample_mode=downsample_mode, max_new_tokens=512)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])

Video Inference

messages = [
    {
        "role": "user",
        "content": [
            {"type": "video", "url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/football.mp4"},
            {"type": "text", "text": "Describe this video in detail. Follow the timeline and focus on on-screen text, interface changes, main actions, and scene changes."},
        ],
    }
]

downsample_mode = "16x"  # Using `downsample_mode="4x"` for Finer Detail

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
    downsample_mode=downsample_mode,
    max_num_frames=128,
    stack_frames=1,
    max_slice_nums=1,
    use_image_id=False,
).to(model.device)

generated_ids = model.generate(**inputs, downsample_mode=downsample_mode, max_new_tokens=2048)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])

Advanced Parameters

You can customize image/video processing by passing additional parameters to apply_chat_template:

Note: downsample_mode must be passed to both apply_chat_template (for correct placeholder count) and generate (for the vision encoder). All other parameters only need to be passed to apply_chat_template.

Serving with `transformers serve`

Hugging Face Transformers includes a lightweight OpenAI-compatible server for quick testing and moderate-load deployment.

pip install "transformers[serving]>=5.7.0"

Start the server:

transformers serve openbmb/MiniCPM-V-4.6 --port 8000 --host 0.0.0.0 --continuous-batching

Send a request:

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "openbmb/MiniCPM-V-4.6",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"}},
        {"type": "text", "text": "What causes this phenomenon?"}
      ]
    }]
  }'

Deploy MiniCPM-V 4.6 on iOS, Android, and HarmonyOS Platforms

We open-source edge deployment guidance of all these platforms, so that developers can deploy on their own devices in a few steps. Try the apps via the download page, or follow the edge deployment guide for the full source.

Use MiniCPM-V 4.6 in Other Inference and Training Frameworks

MiniCPM-V 4.6 supports inference frameworks including SGLang, vLLM, llama.cpp, Ollama, and training frameworks including LLaMA-Factory, SWIFT.

Acknowledgements Copy Section

Click to view acknowledgements.

We would like to thank the following projects:

• Qwen3.5 for providing language backbone
• SigLIP2 for providing vision understanding module
• Transformers

Supported Inference and Training Frameworks and API Copy Chapter

Inference: vLLM, SGLang, llama.cpp, Ollama Copy Section

MiniCPM-V 4.6 supports inference with vLLM, SGLang, llama.cpp and Ollama. See the deployment guides below, or refer to our Cookbook for more details.

Training: LLaMA-Factory, SWIFT Copy Section

MiniCPM-V 4.6 supports fine-tuning with LLaMA-Factory and SWIFT. Refer to our Cookbook for more details.

API Copy Section

Refer to the API Guide for MiniCPM-V 4.6.

Model Zoo Copy Chapter

Minicpm V4dot6 Zh (`docs/minicpm_v4dot6_zh.md`) Copy Section

MiniCPM-V 4.6 Copy Chapter

归档日期：2026-10-10

English | 简体中文 | 返回 README

在线体验

MiniCPM-V 4.6 是 MiniCPM-V 系列的最新模型，也是迄今最端侧友好的一代模型。该模型基于 SigLIP2-400M 和 Qwen3.5-0.8B LLM 构建。其延续了 MiniCPM-V 在单图、多图和视频理解方面的强大能力，同时显著提升了计算效率，还首次支持了 4倍/16倍 混合视觉 token 压缩率，其主要特点包括：

• 🔥 领先的基础能力。
  MiniCPM-V 4.6 在 Artificial Analysis Intelligence Index 基准上获得 13 分，以 19 倍更低的 token 成本超过 Qwen3.5-0.8B 的 10 分，并以 43 倍更低的 token 成本超过 Qwen3.5-0.8B-Thinking 的 11 分。同时，它也超过了参数规模更大的 Ministral 3 3B（11 分）。
• 💪 出色的多模态能力。
  MiniCPM-V 4.6 在绝大多数图文理解任务上优于 Qwen3.5-0.8B，并在 OpenCompass、RefCOCO、HallusionBench、MUIRBench、OCRBench 等众多评测基准上展现出 Qwen3.5 2B 级别的能力。
• 🚀 极致高效架构。
  MiniCPM-V 4.6 基于 LLaVA-UHD v4，引入 ViT 内部视觉 token 早压缩机制，将视觉编码阶段计算量降低 50% 以上，在效率上甚至超越部分更小的模型，相比 Qwen3.5-0.8B 实现约 1.5 倍的 token 吞吐；同时支持 4 倍/16 倍混合视觉 token 压缩率，在精度与速度之间灵活切换。
• 📱 广泛的手机平台支持。
  MiniCPM-V 4.6 可在 iOS、安卓、鸿蒙三大主流手机平台完成部署，并开源配备了端侧适配代码，开发者可在自己的设备上一键复现端侧体验。
• 🛠️ 开发者友好。
  MiniCPM-V 4.6 适配 SGLang、vLLM、llama.cpp、Ollama 等推理框架，并支持 SWIFT、LLaMA-Factory 等微调生态。开发者可以在消费级显卡上为新领域、新任务快速定制模型。我们还提供了覆盖 GGUF、BNB、AWQ、GPTQ 格式的多种量化版本权重，适配多样的部署需求。

性能评估 Copy Section

综合性能（Instruct）

点击查看 MiniCPM-V 4.6-Thinking 的综合性能。

MiniCPM-V 4.6 推理效率

典型示例 Copy Section

MiniCPM-V 4.6 可以在 iOS、安卓、鸿蒙 等主流端侧平台完成部署。

使用说明 Copy Section

基于 Transformers 推理

点击展开基于 Transformers 的推理示例。

安装

pip install "transformers[torch]>=5.7.0" torchvision torchcodec

CUDA 兼容性提示： torchcodec（用于视频解码）可能与部分 CUDA 版本存在兼容性问题。例如 torch>=2.11 默认使用 CUDA 13.1，在 CUDA 12.x 环境下可能出现 RuntimeError: Could not load libtorchcodec 等错误。两种解决方案：

1. 用 PyAV 替代 torchcodec —— 图像和视频推理均可正常使用，无 CUDA 版本限制：
   bash Copy

   pip install "transformers[torch]>=5.7.0" torchvision av

2. 安装 torch 时指定 CUDA 版本以匹配当前环境（如 CUDA 12.8）：
   bash Copy

   pip install "transformers>=5.7.0" torchvision torchcodec --index-url https://download.pytorch.org/whl/cu128

加载模型

from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "openbmb/MiniCPM-V-4.6"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id, torch_dtype="auto", device_map="auto"
)

# 推荐使用 Flash Attention 2 以获得更好的加速与显存节省，
# 尤其在多图和视频场景下效果显著。
# model = AutoModelForImageTextToText.from_pretrained(
#     model_id,
#     torch_dtype=torch.bfloat16,
#     attn_implementation="flash_attention_2",
#     device_map="auto",
# )

图片推理

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"},
            {"type": "text", "text": "What causes this phenomenon?"},
        ],
    }
]

downsample_mode = "16x"  # Using `downsample_mode="4x"` for Finer Detail

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
    downsample_mode=downsample_mode,
    max_slice_nums=36,
).to(model.device)

generated_ids = model.generate(**inputs, downsample_mode=downsample_mode, max_new_tokens=512)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])

视频推理

messages = [
    {
        "role": "user",
        "content": [
            {"type": "video", "url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/football.mp4"},
            {"type": "text", "text": "Describe this video in detail. Follow the timeline and focus on on-screen text, interface changes, main actions, and scene changes."},
        ],
    }
]

downsample_mode = "16x"  # Using `downsample_mode="4x"` for Finer Detail

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
    downsample_mode=downsample_mode,
    max_num_frames=128,
    stack_frames=1,
    max_slice_nums=1,
    use_image_id=False,
).to(model.device)

generated_ids = model.generate(**inputs, downsample_mode=downsample_mode, max_new_tokens=2048)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])

高级参数

可以通过向 apply_chat_template 传递额外参数来自定义图片/视频处理行为：

注意： downsample_mode 需要同时传给 apply_chat_template（确保占位符数量正确）和 generate（控制视觉编码器行为）。其他参数只需传给 apply_chat_template。

使用 `transformers serve` 部署服务

Hugging Face Transformers 内置了一个轻量级 OpenAI 兼容服务器，适合快速测试和中等负载部署。

pip install "transformers[serving]>=5.7.0"

启动服务：

transformers serve openbmb/MiniCPM-V-4.6 --port 8000 --host 0.0.0.0 --continuous-batching

发送请求：

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "openbmb/MiniCPM-V-4.6",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"}},
        {"type": "text", "text": "What causes this phenomenon?"}
      ]
    }]
  }'

iOS、安卓、鸿蒙端侧平台推理

我们将全部端侧适配代码开源，方便开发者在自己的设备上一键复现。可通过下载页体验，或参考端侧部署仓库获取完整源码。

在其他训练、推理框架中使用 MiniCPM-V 4.6

MiniCPM-V 4.6 支持 SGLang, vLLM, llama.cpp, Ollama 等推理框架，和 LLaMA-Factory, SWIFT 等训练框架。

致谢 Copy Section

点击查看致谢。

我们对下列项目表示衷心感谢：

• Qwen3.5 提供了语言基座
• SigLIP2 提供了视觉理解模块
• Transformers

训练和推理框架与 API 支持 Copy Chapter

推理：vLLM、SGLang、llama.cpp、Ollama Copy Section

MiniCPM-V 4.6 推理适配 vLLM、SGLang、llama.cpp、Ollama 等框架。可点击下表查看部署指南，或参考我们的使用指南。

训练：LLaMA-Factory、SWIFT Copy Section

MiniCPM-V 4.6 支持通过 LLaMA-Factory 和 SWIFT 等框架训练。具体用法可以参考我们的使用指南。

API Copy Section

MiniCPM-V 4.6 的 API 用法见 API Guide。

模型库 Copy Chapter

Minicpm V4dot7 En (`docs/minicpm_v4dot7_en.md`) Copy Section

← Home · 中文

MiniCPM-V 4.7

On-device results · Evaluation · Examples · Usage · Model weights

MiniCPM-V 4.7 is the latest model in the MiniCPM-V series, available in 35BA3B and 1B variants, with 35BA3B being the first MoE model in the MiniCPM-V series. It builds on the family's single-image, multi-image, and video understanding capabilities, combining stronger foundation capabilities with efficient visual encoding and on-device inference. Notable features of MiniCPM-V 4.7 include:

• 📱 The First Open-Source MoE Model with Lossless, Smooth On-Device Inference.
  With our Lookahead Expert Scheduling (LES) technique introduced alongside MiniCPM-V 4.7, combined with fine-grained layer-wise quantization and DSpark speculative sampling integration, MiniCPM-V 4.7 35BA3B becomes the first open-source MoE model to run smoothly on-device without compromising performance, achieving a decoding speed of 13+ tokens/s with just 3 GB of memory on iPhone 17 Pro Max. We also provide DSpark speculative decoding heads for 1B (DSpark-BF16, DSpark-GGUF) and 35B-A3B (DSpark-BF16, DSpark-GGUF).
• 🔥 A New Pareto Frontier in Performance and Efficiency.
  As model capabilities continue to advance, inference efficiency has become an important measure of a model's value. On resource-constrained devices, delivering the strongest possible capabilities within a limited resource budget calls for higher capability density. The 35BA3B and 1B variants of MiniCPM-V 4.7 each advance the performance–efficiency Pareto frontier for MLLMs. In Instruct mode, 35BA3B achieves a higher overall score in less inference time than Qwen3.6 35BA3B; the 1B variant continues to improve foundation capabilities over MiniCPM-V 4.6 and substantially outperforms Qwen3.5 0.8B in both overall score and inference efficiency.
• 🚀 Efficient Architecture.
  Based on the intra-ViT early compression technique from LLaVA-UHD v4, MiniCPM-V 4.7 compresses visual tokens early within the vision encoder, reducing computation in subsequent encoder layers. It also supports mixed 4x/16x visual token compression, allowing developers to balance visual detail and inference cost across tasks.
• 💻 Broad Platform and Chip Support.
  MiniCPM-V 4.7 supports iOS, Android, and HarmonyOS, together with open-source on-device deployment support. It also provides day-0 support for chip platforms from Rockchip, Houmo, Intel, AMD, and Arm, extending deployment options across mobile and local computing devices.
• 🛠️ Developer Friendly.
  MiniCPM-V 4.7 supports inference frameworks including SGLang, vLLM, llama.cpp, and Ollama, along with fine-tuning frameworks including SWIFT and LLaMA-Factory. It provides GGUF, BNB, AWQ, and GPTQ quantized variants and adds FP8 quantization support. We also release Self-Replay Fine-Tuning (SRFT), a fine-tuning approach that helps developers extend model capabilities at low cost while reducing forgetting of existing capabilities, making further customization easier.

On-Device Capability and Efficiency Copy Section

Smooth On-Device MoE Inference without compromising performance

An MoE model can have far more total parameters than it activates for a single token, but its on-device storage requirements are not limited to those active parameters. Different tokens can select different experts at different layers, while device memory often cannot hold all expert weights. Without a latency constraint, weights can remain on SSD or other storage, be loaded as needed, and be released after computation. This makes large-model inference possible, but repeated loading stalls can make decoding too slow for real-time interaction.

Another approach is to restrict the pool of available experts. For example, fixing an expert subset for an entire prompt can reduce loading frequency, but limits subsequent token-level routing to that subset, potentially excluding experts the model would otherwise select and reducing its capabilities. The central goal is therefore smooth inference without compromising performance: preserving model capability while delivering sufficient throughput under limited memory and bandwidth.

MiniCPM-V 4.7 35BA3B addresses this challenge through three complementary techniques:

• Lookahead Expert Scheduling (LES) uses cross-layer routing prediction to preload experts likely to be needed by the next layer. Among the experts selected by the actual router, it prioritizes those successfully prefetched and overlaps their computation with loading the remaining required experts. Predictions guide only prefetching and scheduling; computation still follows the model's original token-level routing, and required experts missed by the prediction are loaded and evaluated as usual.
• Fine-grained layer-wise quantization reduces the storage footprint and memory traffic of expert weights while preserving model capability, making expert loading better suited to on-device memory and bandwidth constraints.
• DSpark speculative sampling integration incorporates speculative sampling into the on-device inference process, working with expert scheduling and quantization to improve decoding efficiency.

Together, these techniques make MiniCPM-V 4.7 35BA3B the first open-source MoE model to run smoothly on-device without compromising performance, achieving a decoding speed of 13+ tokens/s with just 3 GB of memory on iPhone 17 Pro Max.

On-Device MoE Inference: Before and After Optimization

Evaluation Copy Section

Significantly Advancing the Performance–Efficiency Pareto Frontier

The 35BA3B and 1B variants of MiniCPM-V 4.7 each advance the performance–efficiency Pareto frontier. In Instruct mode, 35BA3B reaches a higher overall score in less inference time than Qwen3.6 35BA3B, and 1B substantially outperforms Qwen3.5 0.8B in both overall score and inference efficiency.

Overall Performance (35BA3B, Instruct)

Vision-Language

Language

Overall Performance (1B, Instruct)

Examples Copy Section

MiniCPM-V 4.7 can run on phones and tablets. The clips below are unedited screen recordings captured directly on the devices.

Usages Copy Section

Inference with Transformers

Click to expand the Transformers inference examples.

Installation

pip install "transformers[torch]>=5.18.0" torchvision torchcodec

Note on CUDA compatibility: torchcodec (used for video decoding) may have compatibility issues with certain CUDA versions. For example, torch>=2.11 bundles CUDA 13.1 by default, while environments with CUDA 12.x may encounter errors such as RuntimeError: Could not load libtorchcodec. Two workarounds:

1. Replace torchcodec with PyAV — supports both image and video inference without CUDA version constraints:
   bash Copy

   pip install "transformers[torch]>=5.18.0" torchvision av

2. Pin the CUDA version when installing torch to match your environment (e.g. CUDA 12.8):
   bash Copy

   pip install "transformers>=5.18.0" torchvision torchcodec --index-url https://download.pytorch.org/whl/cu128

Load the Model

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "openbmb/MiniCPM-V-4.7-35B-A3B"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id, torch_dtype="auto", device_map="auto"
)

# Flash Attention 2 is recommended for faster inference and lower GPU memory usage,
# especially in multi-image and video scenarios.
# model = AutoModelForImageTextToText.from_pretrained(
#     model_id,
#     torch_dtype=torch.bfloat16,
#     attn_implementation="flash_attention_2",
#     device_map="auto",
# )

Image Inference

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"},
            {"type": "text", "text": "What causes this phenomenon?"},
        ],
    }
]

downsample_mode = "16x"  # Using `downsample_mode="4x"` for Finer Detail

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
    downsample_mode=downsample_mode,
    max_slice_nums=36,
    enable_thinking=False,
).to(model.device)

generated_ids = model.generate(**inputs, downsample_mode=downsample_mode, max_new_tokens=512)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])

Video Inference

messages = [
    {
        "role": "user",
        "content": [
            {"type": "video", "url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/football.mp4"},
            {"type": "text", "text": "Describe this video in detail. Follow the timeline and focus on on-screen text, interface changes, main actions, and scene changes."},
        ],
    }
]

downsample_mode = "16x"  # Using `downsample_mode="4x"` for Finer Detail

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
    downsample_mode=downsample_mode,
    max_num_frames=128,
    stack_frames=1,
    max_slice_nums=1,
    use_image_id=False,
    enable_thinking=False,
).to(model.device)

generated_ids = model.generate(**inputs, downsample_mode=downsample_mode, max_new_tokens=2048)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])

Advanced Parameters

You can customize image/video processing by passing additional parameters to apply_chat_template:

Note: downsample_mode must be passed to both apply_chat_template (for correct placeholder count) and generate (for the vision encoder). All other parameters only need to be passed to apply_chat_template.

Serving with `transformers serve`

Hugging Face Transformers includes a lightweight OpenAI-compatible server for quick testing and moderate-load deployment.

pip install "transformers[serving]>=5.18.0"

Start the server:

transformers serve openbmb/MiniCPM-V-4.7-35B-A3B --port 8000 --host 0.0.0.0

Send a request:

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "openbmb/MiniCPM-V-4.7-35B-A3B",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"}},
        {"type": "text", "text": "What causes this phenomenon?"}
      ]
    }]
  }'

Tool calling example:

curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "openbmb/MiniCPM-V-4.7-35B-A3B",
  "messages": [{"role": "user", "content": [
    {"type": "text", "text": "the weather of Beijing"}
  ]}],
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Get the current weather for a given location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {"type": "string", "description": "City name"}
        },
        "required": ["location"]
      }
    }
  }]
}'

Recommended Sampling Parameters

We recommend using the following set of sampling parameters for generation in instruct mode: temperature=0.7, top_p=0.95, top_k=100, min_p=0.0, presence_penalty=0, repetition_penalty=1.03.

Speculative decoding (DSpark)

Click to show speculative decoding.

We release dedicated DSpark draft models for MiniCPM-V 4.7 35B-A3B and 1B to accelerate inference through speculative decoding. Both heads are available in BF16 and GGUF; see Model Weights for each model and its corresponding DSpark downloads.

Inference with OpenVINO™

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference on Intel CPU/GPU/NPU platforms. OpenVINO provides day-0 support for MiniCPM-V-4.7 on Intel platforms with high-performance inference. See the OpenVINO™ Toolkit to get started. You can install and run the example with this notebook.

On-Device Deployment

MiniCPM-V 4.7 supports deployment on iOS, Android, and HarmonyOS, with day-0 support for Rockchip, Houmo, Intel, AMD, and Arm chip platforms. See the edge deployment repository for the apps and source code.

Inference and Fine-Tuning

For supported inference and training frameworks, see Supported Inference and Training Frameworks and API. The MiniCPM-V 4.7 Cookbook covers deployment, fine-tuning, and quantization workflows.

Task guides

• Single-image chat
• Multi-image chat
• Video understanding
• OCR
• PDF parsing
• Visual grounding

Quantization

Acknowledgements Copy Section

Click to view acknowledgements.

We would like to thank the following projects:

• Qwen3.5 and Qwen3.6 for providing the language backbone
• SigLIP2 for providing the vision understanding module
• Transformers

Model Weights Copy Chapter

MiniCPM-V 4.7 is available in 35B-A3B and 1B variants. Each row lists the model weights alongside its corresponding DSpark speculative decoding heads, available in BF16 and GGUF.

The 1B int4 variant uses bitsandbytes; AWQ and GPTQ use W4A16 quantization. Quantized variants reduce memory usage; GGUF weights support CPU inference.

Also available on Ollama.

Minicpm V4dot7 Zh (`docs/minicpm_v4dot7_zh.md`) Copy Section

← 首页 · English

MiniCPM-V 4.7

端侧结果 · 性能评估 · 典型示例 · 使用说明 · 模型权重

MiniCPM-V 4.7 是 MiniCPM-V 系列的最新模型，提供 35BA3B 和 1B 两种规格，其中 35BA3B 是 MiniCPM-V 系列首个 MoE 模型。它延续了系列模型的单图、多图和视频理解能力，在提升基础能力的同时，保持高效的视觉编码与端侧推理，其主要特点包括：

• 📱 首个端侧无损流畅运行的开源 MoE 模型。
  基于随 MiniCPM-V 4.7 同步提出的自研 LES（前瞻专家调度） 技术，配合精细的层级量化与 DSpark 投机采样适配，MiniCPM-V 4.7 35BA3B 成为首个支持端侧无损流畅运行的开源 MoE 模型，在 iPhone 17 Pro Max 上以仅 3GB 内存达到 13+ token/s 的解码速度。我们也适配了 DSpark 投机采样，并提供 1B（DSpark-BF16、DSpark-GGUF）和 35B-A3B（DSpark-BF16、DSpark-GGUF）的投机采样头。
• 🔥 性能与效率的新帕累托前沿。
  随着模型能力不断发展，推理效率也成为衡量模型价值的重要维度；而在资源受限的端侧，更需要在有限的开销内提供尽可能强的能力，追求更高的能力密度。MiniCPM-V 4.7 以 35BA3B 和 1B 分别同时刷新了多模态大模型的性能—效率的帕累托前沿。在 Instruct 模式下，35BA3B 以更短的推理时间取得比 Qwen3.6 35BA3B 更高的综合得分；1B 版本相比 MiniCPM-V 4.6 持续提升基础能力，并在综合得分和推理效率上均显著领先 Qwen3.5 0.8B。
• 🚀 高效架构。
  MiniCPM-V 4.7 基于 LLaVA-UHD v4 的 ViT 内提前压缩技术，在视觉编码器内部提前压缩视觉 token，减少后续编码层的计算开销；同时支持 4 倍/16 倍混合视觉 token 压缩率，可根据任务需求在视觉细节与推理开销之间灵活取舍。
• 💻 多平台与芯片支持。
  MiniCPM-V 4.7 同时支持 iOS、安卓和鸿蒙，并提供开源端侧适配代码。同时，首日支持瑞芯微、后摩、Intel、AMD 和 Arm 等芯片平台，覆盖更多移动设备与本地计算场景。
• 🛠️ 开发者友好。
  MiniCPM-V 4.7 适配 SGLang、vLLM、llama.cpp、Ollama 等推理框架，支持 SWIFT、LLaMA-Factory 等微调框架，提供 GGUF、BNB、AWQ、GPTQ 等量化版本，并新增 FP8 量化支持。同步公开 Self-Replay Fine-Tuning (SRFT) 微调方案，帮助开发者低成本扩展模型能力、减少原有能力遗忘，方便二次开发。

端侧能力与效率 Copy Section

无损且流畅的 MoE 端侧推理

MoE 的总参数量可以远大于单个 token 激活的参数量，但这并不意味着端侧推理只需承担激活参数的存储成本。不同 token 在不同层可能选择不同的专家，而设备内存通常无法容纳全部专家权重。如果不约束时延，可以把权重保存在 SSD 等存储中，计算时按需载入、用后释放，完成大模型推理；频繁加载权重带来的等待，却会使解码速度难以满足实时交互的需求。

另一种办法是缩小可供选择的专家集合。例如，按整段 prompt 固定专家子集，可以降低加载频率，但会将后续 token 的动态路由限制在该子集内，可能排除模型原本会选择的专家，带来能力损失。因此，端侧 MoE 的核心目标是同时做到无损和流畅：在保留模型能力的前提下，解决有限内存与带宽下的推理吞吐问题。

MiniCPM-V 4.7 35BA3B 通过三项技术协同实现这一目标：

• LES（前瞻专家调度）：通过跨层路由预测提前加载下一层可能用到的专家，优先计算实际路由选中且预加载命中的专家，并利用这段计算时间并行加载其余所需专家，减少加载等待。预测仅用于预取与调度，实际计算仍遵循模型原有的 token 级路由；预测未命中的所需专家也会正常加载并参与计算。
• 精细的层级量化：降低专家权重的存储占用与访存开销，在保持模型能力的前提下，使专家加载更适合端侧的内存与带宽条件。
• DSpark 投机采样适配：将投机采样与端侧推理流程结合，与专家调度和量化协同提高解码效率。

这套完整方案使 MiniCPM-V 4.7 35BA3B 成为首个端侧无损流畅运行的开源 MoE 模型，在 iPhone 17 Pro Max 上以仅 3GB 内存达到 13+ token/s 的解码速度。

MoE 端侧推理优化前后对比

性能评估 Copy Section

显著推进性能与推理效率的帕累托前沿

MiniCPM-V 4.7 的 35BA3B 与 1B 同时刷新了性能—效率帕累托前沿。Instruct 模式下，35BA3B 以更短的推理时间超过 Qwen3.6 35BA3B，1B 在综合得分和推理效率上均显著领先 Qwen3.5 0.8B。

完整性能对比（35BA3B，Instruct）

Vision-Language

Language

完整性能对比（1B，Instruct）

典型示例 Copy Section

MiniCPM-V 4.7 可以部署在不同的端侧设备上——手机和平板。以下片段为设备上未经剪辑的原始屏幕录制。

使用说明 Copy Section

基于 Transformers 推理

点击展开基于 Transformers 的推理示例。

安装

pip install "transformers[torch]>=5.18.0" torchvision torchcodec

CUDA 兼容性提示： torchcodec（用于视频解码）可能与部分 CUDA 版本存在兼容性问题。例如 torch>=2.11 默认使用 CUDA 13.1，在 CUDA 12.x 环境下可能出现 RuntimeError: Could not load libtorchcodec 等错误。两种解决方案：

1. 用 PyAV 替代 torchcodec —— 图像和视频推理均可正常使用，无 CUDA 版本限制：
   bash Copy

   pip install "transformers[torch]>=5.18.0" torchvision av

2. 安装 torch 时指定 CUDA 版本以匹配当前环境（如 CUDA 12.8）：
   bash Copy

   pip install "transformers>=5.18.0" torchvision torchcodec --index-url https://download.pytorch.org/whl/cu128

加载模型

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "openbmb/MiniCPM-V-4.7-35B-A3B"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id, torch_dtype="auto", device_map="auto"
)

# 推荐使用 Flash Attention 2 以获得更好的加速与显存节省，
# 尤其在多图和视频场景下效果显著。
# model = AutoModelForImageTextToText.from_pretrained(
#     model_id,
#     torch_dtype=torch.bfloat16,
#     attn_implementation="flash_attention_2",
#     device_map="auto",
# )

图片推理

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"},
            {"type": "text", "text": "What causes this phenomenon?"},
        ],
    }
]

downsample_mode = "16x"  # Using `downsample_mode="4x"` for Finer Detail

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
    downsample_mode=downsample_mode,
    max_slice_nums=36,
    enable_thinking=False,
).to(model.device)

generated_ids = model.generate(**inputs, downsample_mode=downsample_mode, max_new_tokens=512)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])

视频推理

messages = [
    {
        "role": "user",
        "content": [
            {"type": "video", "url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/football.mp4"},
            {"type": "text", "text": "Describe this video in detail. Follow the timeline and focus on on-screen text, interface changes, main actions, and scene changes."},
        ],
    }
]

downsample_mode = "16x"  # Using `downsample_mode="4x"` for Finer Detail

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
    downsample_mode=downsample_mode,
    max_num_frames=128,
    stack_frames=1,
    max_slice_nums=1,
    use_image_id=False,
    enable_thinking=False,
).to(model.device)

generated_ids = model.generate(**inputs, downsample_mode=downsample_mode, max_new_tokens=2048)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])

高级参数

可以通过向 apply_chat_template 传递额外参数来自定义图片/视频处理行为：

注意： downsample_mode 需要同时传给 apply_chat_template（确保占位符数量正确）和 generate（控制视觉编码器行为）。其他参数只需传给 apply_chat_template。

使用 `transformers serve` 部署服务

Hugging Face Transformers 内置了一个轻量级 OpenAI 兼容服务器，适合快速测试和中等负载部署。

pip install "transformers[serving]>=5.18.0"

启动服务：

transformers serve openbmb/MiniCPM-V-4.7-35B-A3B --port 8000 --host 0.0.0.0

发送请求：

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "openbmb/MiniCPM-V-4.7-35B-A3B",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"}},
        {"type": "text", "text": "What causes this phenomenon?"}
      ]
    }]
  }'

工具调用示例:

curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "openbmb/MiniCPM-V-4.7-35B-A3B",
  "messages": [{"role": "user", "content": [
    {"type": "text", "text": "the weather of Beijing"}
  ]}],
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Get the current weather for a given location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {"type": "string", "description": "City name"}
        },
        "required": ["location"]
      }
    }
  }]
}'

推荐采样参数

我们推荐在 instruct mode 下使用以下采样参数进行生成：temperature=0.7、top_p=0.95、top_k=100、min_p=0.0、presence_penalty=0、repetition_penalty=1.03。

投机解码 DSpark

点击展开投机解码。

我们分别为 MiniCPM-V 4.7 35B-A3B 和 1B 发布了专门训练的 DSpark 草稿模型，通过投机解码加速推理。两种规格均提供 BF16 和 GGUF 版本，主模型及其对应的 DSpark 下载链接见模型权重。

基于 OpenVINO™ 推理

OpenVINO™ 是一个用于在 Intel CPU/GPU/NPU 平台上优化和部署 AI 推理的开源工具包。OpenVINO 首日支持 MiniCPM-V-4.7 在 Intel 平台上的高性能推理。可参考 OpenVINO™ Toolkit 快速入门，并通过 该 notebook 安装和运行示例。

端侧部署

MiniCPM-V 4.7 支持 iOS、安卓、鸿蒙部署，并首日支持瑞芯微、后摩、Intel、AMD 和 Arm 等芯片平台。应用与源码可参考端侧部署仓库。

推理与微调

支持的推理和训练框架见训练和推理框架与 API 支持。MiniCPM-V 4.7 使用手册提供部署、微调与量化等使用指南。

场景教程

• 单图问答
• 多图问答
• 视频理解
• OCR
• PDF 解析
• 视觉定位

量化

致谢 Copy Section

点击查看致谢。

我们对下列项目表示衷心感谢：

• Qwen3.5 和 Qwen3.6 提供了语言基座
• SigLIP2 提供了视觉理解模块
• Transformers

模型权重 Copy Chapter

MiniCPM-V 4.7 提供 35B-A3B 和 1B 两种规格。每行列出主模型权重及其对应的 DSpark 投机采样头，DSpark 均提供 BF16 和 GGUF 版本。