## File: README.md * 此开源项目旨在完全从 0 开始，仅用 3 块钱成本与 2 小时训练时间，即可训练出规模约为 64M 的超小语言模型 MiniMind。 * MiniMind 系列极其轻量，主线最小版本体积约为 GPT-3 的 $\frac{1}{2700}$，力求让普通个人 GPU 也能快速完成训练与复现。 * 项目同时开源了大模型的极简结构与完整训练链路，覆盖 MoE、数据清洗、预训练（Pretrain）、监督微调（SFT）、LoRA、RLHF（DPO）、RLAIF（PPO / GRPO / CISPO）、Tool Use、Agentic RL、自适应思考与模型蒸馏等全过程代码。 * MiniMind 同时拓展了视觉模态模型 [MiniMind-V](https://github.com/jingyaogong/minimind-v)、多模态 Omni 模型 [MiniMind-O](https://github.com/jingyaogong/minimind-o)、扩散语言模型（MiniMind-dLM）、线性模型（MiniMind-Linear），详见 [Discussion](https://github.com/jingyaogong/minimind/discussions)。 * 项目所有核心算法代码均从 0 使用 PyTorch 原生实现，不依赖第三方库提供的高层抽象接口。 * 这不仅是一个大语言模型全阶段开源复现项目，也是一套面向 LLM 入门与实践的教程。 * 希望此项目能为更多人提供一个可复现、可理解、可扩展的起点，一起感受创造的乐趣，并推动更广泛 AI 社区的进步。 > 注：本项目基于 Apache 2.0 协议开源，完全免费。“2 小时” 指 SFT 阶段在单张 NVIDIA 3090 上跑完 `1 epoch` 的实测耗时，“3 块钱” 指对应时段的 GPU 租用成本。 --- --- # 📌 项目介绍 大语言模型（Large Language Model, LLM）的出现，引发了全球范围内对 AI 的空前关注。无论是 ChatGPT、DeepSeek 还是 Qwen，都以惊艳的效果让人真切感受到这场技术浪潮的冲击力。然而，动辄数百亿参数的模型规模，使得它们对个人设备而言不仅难以训练，甚至连部署都显得遥不可及。打开大模型的“黑盒子”，真正去理解其内部运作机制，本应是一件令人心潮澎湃的事。遗憾的是，绝大多数探索最终都止步于使用 LoRA 等技术对现有大模型做少量微调，学习一些新指令或特定任务。这更像是在教牛顿如何使用 21 世纪的智能手机——虽然有趣，却偏离了理解物理本质的初衷。 与此同时，第三方的大模型框架与工具库，如 `transformers` / `trl` / `peft` 等，往往只暴露出高度抽象的接口。只需短短十几行代码，就可以完成“加载模型 + 加载数据集 + 推理 + 强化学习”的全流程训练。这种高效封装固然便利，却也在一定程度上把开发者与底层实现隔离开来，削弱了深入理解 LLM 核心代码的机会。我认为 “用乐高自己拼出一架飞机，远比坐在头等舱里飞行更让人兴奋”，然而更现实的问题是，互联网上充斥着大量付费课程和营销内容，用漏洞百出、一知半解的讲解包装所谓的 AI 教程。正因如此，本项目的初衷就是尽可能降低 LLM 的学习门槛，让每个人都能从理解每一行代码开始，从 0 开始亲手训练一个极小的语言模型。是的，从**零开始训练**，而不是仅仅停留在**推理**层面。最低只需不到 3 块钱的服务器成本，就能亲身体验从 0 到 1 构建一个语言模型的全过程。 😊 一起感受创造的乐趣吧！ --- #### 🎉 本项目包含以下内容 - 提供完整的 MiniMind-LLM 结构代码（Dense + MoE），当前主线结构对齐 `Qwen3 / Qwen3-MoE` 生态。 - 提供 Tokenizer 与分词器训练代码，支持 ``、``、`` 等模板标记。 - 覆盖 Pretrain、SFT、LoRA、RLHF-DPO、RLAIF（PPO / GRPO / CISPO）、Tool Use、Agentic RL、自适应思考与模型蒸馏等完整训练流程。 - 提供全阶段开源数据，覆盖收集、蒸馏、清洗与去重后的高质量数据集。 - 关键训练算法与核心模块均从 0 实现，不依赖第三方框架封装。 - 兼容 `transformers`、`trl`、`peft` 等主流框架，以及 `llama.cpp`、`vllm`、`ollama` 等常用推理引擎与 `Llama-Factory` 等训练框架。 - 支持单机单卡与单机多卡（DDP、DeepSpeed）训练，支持 wandb / swanlab 可视化与动态启停训练。 - 支持在 C-Eval、C-MMLU、OpenBookQA 等第三方测评集上进行评测，并支持通过 YaRN 实现 RoPE 长文本外推。 - 提供兼容 OpenAI API 协议的极简服务端，便于接入 FastGPT、Open-WebUI 等第三方 Chat UI，并支持 `reasoning_content`、`tool_calls`、`open_thinking`。 - 提供基于 Streamlit 的极简聊天 WebUI，支持思考展示、工具选择与多轮 Tool Call。 - 包含实验性拓展：离散扩散语言模型（[dLM](https://github.com/jingyaogong/minimind/discussions/618)）与线性注意力模型（[Linear Attention](https://github.com/jingyaogong/minimind/discussions/704)），均可基于主线 AR 模型进行续训。 #### 🎉 已发布模型列表 | 模型 | 参数量 | Release | |------|--------|---------| | minimind-3 | 64M | 2026.04.01 | | minimind-3-moe | 198M-A64M | 2026.04.01 | | minimind2-small | 26M | 2025.04.26 | | minimind2-moe | 145M | 2025.04.26 | | minimind2 | 104M | 2025.04.26 | | minimind-v1-small | 26M | 2024.08.28 | | minimind-v1-moe | 4×26M | 2024.09.17 | | minimind-v1 | 108M | 2024.09.01 | --- #### 📝 更新日志 **🔥 2026-04-01** - 发布 `minimind-3` / `minimind-3-moe`：结构、Tokenizer、训练链路、推理接口与默认配置全面更新 - 结构主线对齐 `Qwen3 / Qwen3-MoE` 生态：Dense 约 `64M`，MoE 约 `198M-A64M`，并移除了 shared expert 设计 - 默认训练数据切换为 `pretrain_t2t(_mini).jsonl`、`sft_t2t(_mini).jsonl`、`rlaif.jsonl`、`agent_rl.jsonl` 与 `agent_rl_math.jsonl` - 移除独立 `train_reason.py`；思考能力统一由 `chat_template + ` 与 `open_thinking` 自适应开关控制 - `toolcall` 能力已混入 `sft_t2t / sft_t2t_mini` 主线数据，默认 `full_sft` 即具备基础 Tool Call 能力；同时新增 `scripts/chat_api.py` 等推理示例 - 新增原生 `Agentic RL` 训练脚本 `train_agent.py`，支持多轮 Tool-Use 场景下的 `GRPO / CISPO` - RLAIF / Agentic RL 训练流程完成 `rollout engine` 解耦，支持更灵活地切换生成后端 - `serve_openai_api.py` 与 `web_demo.py` 新增 `reasoning_content` / `tool_calls` / `open_thinking` 支持 - Tokenizer 基于 `BPE + ByteLevel` 更新，并新增工具调用与思考标记，预留 buffer token 便于后续扩展 - 新增 LoRA 权重合并导出流程，可通过 `scripts/convert_model.py` 将基础模型与 LoRA 权重合并为新的完整模型权重 - 结构图资源更新，README 大面积更新 **2025-10-24** - 🔥 新增RLAIF训练算法：PPO、GRPO、SPO（从0原生实现） - 新增断点续训功能：支持训练自动恢复、跨GPU数量恢复、wandb记录连续性 - 新增RLAIF数据集：rlaif-mini.jsonl（从SFT数据随机采样1万条）；简化DPO数据集，加入中文数据 - 新增YaRN算法：支持RoPE长文本外推，提升长序列处理能力 - Adaptive Thinking：Reason模型可选是否启用思考链 - chat_template全面支持Tool Calling和Reasoning标签（``、``等） - 新增RLAIF完整章节、训练曲线对比、算法原理折叠说明 - [SwanLab](https://swanlab.cn/)替代WandB（国内访问友好，API完全兼容） - 规范化所有代码 & 修复一些已知bugs **2025-04-26** - 重要更新 - 如有兼容性需要，可访问[🔗旧仓库内容🔗](https://github.com/jingyaogong/minimind/tree/7da201a944a90ed49daef8a0265c959288dff83a)。 - MiniMind模型参数完全改名，对齐Transformers库模型（统一命名）。 - generate方式重构，继承自GenerationMixin类。 - 🔥支持llama.cpp、vllm、ollama等热门三方生态。 - 规范代码和目录结构。 - 改动词表``->`<|im_start|><|im_end|>` ```text 为兼容第三方推理框架llama.cpp、vllm，本次更新需付出一些可观代价。 本次更新不再支持「直接」加载25-04-26以前的旧模型进行推理。 由于Llama位置编码方式与minimind存在区别，导致映射Llama模型后QK值存在差异 minimind2系列旧模型均经过权重映射+（微调训练）QKVO线性层校准恢复而来。 本次更新后将放弃对`minimind-v1`全系列的维护，并在仓库中下线。 ``` **More...** **2025-02-09** - 迎来发布以来重大更新，Release minimind2 Series。 - 代码几乎全部重构，使用更简洁明了的统一结构。 如有旧代码的兼容性需要，可访问[🔗旧仓库内容🔗](https://github.com/jingyaogong/minimind/tree/6e9cd28ef9b34a0a10afbdf6f59e65cb6e628efb)。 - 免去数据预处理步骤。统一数据集格式，更换为`jsonl`格式杜绝数据集下载混乱的问题。 - minimind2系列效果相比MiniMind-V1显著提升。 - 小问题：{kv-cache写法更标准、MoE的负载均衡loss被考虑等等} - 提供模型迁移到私有数据集的训练方案（医疗模型、自我认知样例）。 - 精简预训练数据集，并大幅提升预训练数据质量，大幅缩短个人快速训练所需时间，单卡3090即可2小时复现！ - 更新：LoRA微调脱离peft包装，从0实现LoRA过程；DPO算法从0使用PyTorch原生实现；模型白盒蒸馏原生实现。 - minimind2-DeepSeek-R1系列蒸馏模型诞生！ - minimind2具备一定的英文能力！ - 更新minimind2与第三方模型的基于更多大模型榜单测试性能的结果。 **2024-10-05** - 为MiniMind拓展了多模态能力之---视觉 - 移步孪生项目[minimind-v](https://github.com/jingyaogong/minimind-v)查看详情！ **2024-09-27** - 09-27更新pretrain数据集的预处理方式，为了保证文本完整性，放弃预处理成.bin训练的形式（轻微牺牲训练速度）。 - 目前pretrain预处理后的文件命名为：pretrain_data.csv。 - 删除了一些冗余的代码。 **2024-09-17** - 更新minimind-v1-moe模型 - 为了防止歧义，不再使用mistral_tokenizer分词，全部采用自定义的minimind_tokenizer作为分词器。 **2024-09-01** - 更新minimind-v1 (108M)模型，采用minimind_tokenizer，预训练轮次3 + SFT轮次10，更充分训练，性能更强。 - 项目已部署至ModelScope创空间，可以在此网站上体验： - [🔗ModelScope在线体验🔗](https://www.modelscope.cn/studios/gongjy/minimind) **2024-08-27** - 项目首次开源 --- # 📌 快速开始 本人的软硬件配置（供参考） * CPU: Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz * RAM: 128 GB * GPU: NVIDIA GeForce RTX 3090 (24GB) * 8 * Ubuntu==20.04 * CUDA==12.2 * Python==3.10.16 * [requirements.txt](./requirements.txt) ## 第0步 ```bash # 克隆仓库、安装依赖 git clone --depth 1 https://github.com/jingyaogong/minimind cd minimind && pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple ``` ## Ⅰ 🚀 模型推理 ### 1' 下载模型 在项目根目录： ```bash # 方式1 modelscope download --model gongjy/minimind-3 --local_dir ./minimind-3 # 方式2 git clone https://huggingface.co/jingyaogong/minimind-3 ``` ### 2' CLI 推理 ```bash # 方式1：使用 Transformers 格式模型 python eval_llm.py --load_from ./minimind-3 # 方式2：基于 PyTorch 模型（确保./out目录下有对应权重） python eval_llm.py --load_from ./model --weight full_sft ``` ### 3'（可选）WebUI ```bash # 可能需要`python>=3.10`，安装 `pip install streamlit` # ⚠️ 须先将 transformers 格式模型文件夹复制到 ./scripts/ 目录下（例如：cp -r minimind-3 ./scripts/minimind-3），web_demo 脚本会自动扫描该目录下包含权重文件的子文件夹，如不存在则报错 cd scripts && streamlit run web_demo.py ``` ### 4'（可选）第三方推理框架 ```bash # ollama ollama run jingyaogong/minimind-3 # vllm vllm serve /path/to/model --served-model-name "minimind" ``` ## Ⅱ 🛠️ 模型训练 注：提前确认 Torch 的可用后端 ```python import torch print(torch.cuda.is_available()) ``` 若你计划使用 CUDA 训练，建议先确认当前环境是否已正确识别 GPU。 若 `cuda` 不可用，也仍可根据自身设备选择 `CPU` 或 `MPS` 运行，但训练速度与兼容性会有非常大的差异。 如需安装或更换 PyTorch 版本，可参考 [torch_stable](https://download.pytorch.org/whl/torch_stable.html) 与[链接](https://blog.csdn.net/weixin_45456738/article/details/141029610?ops_request_misc=&request_id=&biz_id=102&utm_term=%E5%AE%89%E8%A3%85torch&utm_medium=distribute.pc_search_result.none-task-blog-2~all~sobaiduweb~default-2-141029610.nonecase&spm=1018.2226.3001.4187) ### 1' 下载数据 从下文提供的[数据集下载链接](https://www.modelscope.cn/datasets/gongjy/minimind_dataset/files) 下载所需数据文件，并放入 `./dataset` 目录 > 当前默认仅需下载 `pretrain_t2t_mini.jsonl` 与 `sft_t2t_mini.jsonl`，即可较快复现 `MiniMind Zero` 对话模型。 如有更多需求，下文提供多种搭配方案，可根据自身任务目标与 GPU 资源灵活选择。 ### 2' 开始训练 💡 检查点暂停续训 所有训练脚本均支持检查点保存。添加 `--from_resume 1` 参数后，即可自动检测并恢复训练进度： ```bash python train_pretrain.py --from_resume 1 python train_full_sft.py --from_resume 1 # ... ``` **断点续训说明：** - 训练过程会自动在 `./checkpoints/` 目录保存完整检查点（模型、优化器、训练进度等） - 检查点文件命名：`<权重名>_<维度>_resume.pth`（如：`full_sft_512_resume.pth`） - 支持跨不同 GPU 数量恢复（自动调整 step） - 支持 wandb 训练记录连续性（自动恢复同一个 run） > 适合长时间训练或不稳定环境，无需担心训练中断导致进度丢失 #### 2.1 预训练（必须） ```bash cd trainer && python train_pretrain.py ``` > 训练后，将得到 `out/pretrain_*.pth` 作为输出权重（其中 `*` 为模型 dimension，默认为 `768`） #### 2.2 指令微调（必须） ```bash cd trainer && python train_full_sft.py ``` > 训练后，将得到 `out/full_sft_*.pth` 作为输出权重（其中 `full` 表示全参数微调） #### 2.3 测试已训练模型（可选） 确保待测试的模型 `*.pth` 文件位于 `./out/` 目录下；也可直接前往[此处](https://www.modelscope.cn/models/gongjy/minimind-3-pytorch/files)下载我已训练好的 `*.pth` 权重。 ```bash python eval_llm.py --weight full_sft ``` > `--weight` 用于指定权重名称前缀，例如 `pretrain`、`full_sft` 等；更多参数可直接参考 `eval_llm.py` 注：其它须知 1、所有训练脚本均基于 PyTorch 原生实现，并支持多卡加速。 2、若你的设备有 `N (N > 1)` 张显卡，可通过以下方式启动单机 `N` 卡训练（DDP，也支持扩展到多机多卡）： ```bash torchrun --nproc_per_node N train_xxx.py ``` 3、可根据需要开启 wandb 记录训练过程。 ```bash ... train_xxx.py --use_wandb ``` `2025` 年 `6` 月后，国内网络环境通常无法直连 WandB。MiniMind 当前默认转为使用 [SwanLab](https://swanlab.cn/) 作为训练可视化工具，其接口与 WandB 基本兼容；通常只需将 `import wandb` 替换为 `import swanlab as wandb`，其余调用方式基本无需改动。 --- # 📌 数据介绍 ## Ⅰ Tokenizer 分词器可以粗略理解成 LLM 使用的一本“词典”，负责把自然语言映射成 token id，再把 token id 解码回文本；项目中也提供了`train_tokenizer.py`作为词表训练示例。不建议重新训练 tokenizer，因为词表和切分规则一旦变化，模型权重、数据格式、推理接口与社区生态的兼容性都会下降，也会削弱模型的传播性。同时，tokenizer 还会影响 PPL 这类按 token 统计的指标，因此跨 tokenizer 比较时，BPB（Bits Per Byte）往往更有参考价值，可参考[这篇](https://skeptric.com/perplexity/)。 对 MiniMind 这类小模型来说，词表大小还会直接影响 embedding 层和输出层的参数占比，因此保持词表精简通常是更合适的取舍。 Tokenizer介绍 第三方强大的开源模型例如 Yi、Qwen2、ChatGLM、Mistral、Llama 3 的 tokenizer 词表长度如下： | Tokenizer模型 | 词表大小 | 来源 | | --- | --- | --- | | Yi | 64,000 | 01万物（中国） | | Qwen2 | 151,643 | 阿里云（中国） | | ChatGLM | 151,329 | 智谱AI（中国） | | Mistral | 32,000 | Mistral AI（法国） | | Llama 3 | 128,000 | Meta（美国） | | MiniMind | 6,400 | 自定义 | > 当前主线为避免历史版本歧义并控制整体体积，统一使用 `minimind_tokenizer`，不再维护 `mistral_tokenizer` 版本。 尽管 `minimind_tokenizer` 的词表只有 `6400`，编解码效率弱于 `qwen2`、`glm` 等更偏中文友好的 tokenizer，但它能显著压缩 embedding 层和输出层的参数占比，更适合 MiniMind 这类小模型的体积约束。 从实际使用效果看，这套 tokenizer 并没有明显带来生僻词解码失败的问题，整体仍然足够稳定可用；因此当前主线训练也统一沿用这套词表，而不再额外分叉维护其他 tokenizer 版本。 ## Ⅱ Pretrain数据 `MiniMind-3` 当前主线预训练数据为 `pretrain_t2t.jsonl` / `pretrain_t2t_mini.jsonl`。 这两份数据已经整理成统一的 `text -> next token prediction` 训练格式，目标是在较小算力下兼顾： - 文本质量； - 长度分布； - 中英混合能力； - 与后续 SFT / Tool Calling / RLAIF 阶段的模板衔接。 数据来源包括但不限于通用文本语料、对话整理语料、蒸馏补充语料，以及各类**宽松开源协议**可用的数据集；主线数据会在清洗、去重、长度控制与格式统一后再进入训练。主要来源包括：[匠数大模型数据集](https://www.modelscope.cn/datasets/deepctrl/deepctrl-sft-data)、[Magpie-Align](https://www.modelscope.cn/organization/Magpie-Align) 等公开数据源。 其中： - `pretrain_t2t_mini.jsonl` 更适合快速复现； - `pretrain_t2t.jsonl` 更适合完整训练 `MiniMind-3` 主线模型。 文件数据格式为 ```jsonl {"text": "如何才能摆脱拖延症？治愈拖延症并不容易，但以下建议可能有所帮助。"} {"text": "清晨的阳光透过窗帘洒进房间，桌上的书页被风轻轻翻动。"} {"text": "Transformer 通过自注意力机制建模上下文关系，是现代大语言模型的重要基础结构。"} ``` ## Ⅲ SFT数据 `MiniMind-3` 当前主线 SFT 数据为 `sft_t2t.jsonl` / `sft_t2t_mini.jsonl`。相比更早期的 `sft_512 / sft_1024 / sft_2048` 方案，当前版本更强调： - 统一模板； - 更适合对话 + 思考标签 + Tool Calling 的混合训练； - 尽量减少数据预处理分叉，降低复现成本。 其数据来源包括但不限于高质量指令跟随数据、公开对话数据、模型蒸馏合成数据，以及协议友好的开源数据集；在进入 `t2t` 主线前，会统一为当前仓库使用的多轮对话格式。当前主线中也包含大量合成数据，例如本人基于 `qwen3-4b` 合成的约 `10w` 条 `tool call` 数据，以及 `qwen3` 系列的 `reasoning` 数据等。其中社区主要来源有：[匠数大模型数据集](https://www.modelscope.cn/datasets/deepctrl/deepctrl-sft-data)、[Magpie-Align](https://www.modelscope.cn/organization/Magpie-Align)、[R1-Distill-SFT](https://www.modelscope.cn/datasets/AI-ModelScope/R1-Distill-SFT)、[COIG](https://huggingface.co/datasets/BAAI/COIG)、[Step-3.5-Flash-SFT](https://huggingface.co/datasets/stepfun-ai/Step-3.5-Flash-SFT) 等。公布版本会确保数据来源与处理链路符合对应开源协议的可传递性约束，并遵守 Apache-2.0、CC-BY-NC-2.0 等相关协议要求。 其中： - `sft_t2t_mini.jsonl`：适合快速训练对话模型； - `sft_t2t.jsonl`：适合完整复现主线版本； - `toolcall` 能力已经并入主线 SFT 数据。 所有 SFT 文件数据格式均为（包含对话数据、Tool Use 数据） ```jsonl { "conversations": [ {"role": "user", "content": "你好"}, {"role": "assistant", "content": "你好！"}, {"role": "user", "content": "再见"}, {"role": "assistant", "content": "再见！"} ] } { "conversations": [ {"role": "system", "content": "# Tools ...", "tools": "[...]"}, {"role": "user", "content": "把'你好世界'翻译成english"}, {"role": "assistant", "content": "", "tool_calls": "[{\"name\":\"translate_text\",\"arguments\":{\"text\":\"你好世界\",\"target_language\":\"english\"}}]"}, {"role": "tool", "content": "{\"translated_text\":\"Hello World\"}"}, {"role": "assistant", "content": "Hello World"} ] } ``` ## Ⅳ RL 数据 `MiniMind` 当前主线 RL 数据为 `dpo.jsonl`。数据抽样自 [DPO-En-Zh-20k](https://huggingface.co/datasets/llamafactory/DPO-En-Zh-20k)。 主线中会将这部分样本统一重组为当前仓库使用的偏好学习格式，用于奖励模型或偏好优化阶段训练；其中 `chosen` 表示更符合偏好的回复，`rejected` 表示相对较差的回复。 其中 `dpo.jsonl` 数据格式为 ```json { "chosen": [ {"content": "Q", "role": "user"}, {"content": "good answer", "role": "assistant"} ], "rejected": [ {"content": "Q", "role": "user"}, {"content": "bad answer", "role": "assistant"} ] } ``` 除此之外，其他 RL 数据与 SFT 数据格式保持一致，通常是从 SFT 数据中按总长度和对话轮次筛选得到，并将最后一个 `assistant` 位置留空，供 rollout 阶段续写使用。 ## Ⅴ MiniMind 训练数据集 > [!NOTE] > 当前主线训练所需的核心数据集已开源，因此无需再自行预处理大规模数据集，避免重复性的数据处理工作。 MiniMind训练数据集下载地址： [ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind_dataset/files) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind_dataset/tree/main) > 无需全部clone，可单独下载所需的文件 将下载的数据集文件放到`./dataset/`目录下（✨为推荐的必须项） ```bash ./dataset/ ├── agent_rl.jsonl (86MB) ├── agent_rl_math.jsonl (18MB) ├── dpo.jsonl (53MB) ├── pretrain_t2t_mini.jsonl (1.2GB, ✨) ├── pretrain_t2t.jsonl (10GB) ├── rlaif.jsonl (24MB, ✨) ├── sft_t2t_mini.jsonl (1.6GB, ✨) └── sft_t2t.jsonl (14GB) ``` 注：各数据集简介 * `agent_rl.jsonl` --Agentic RL 主线训练数据，用于 `train_agent.py` 的多轮 Tool-Use / CISPO / GRPO 训练 * `agent_rl_math.jsonl` --Agentic RL 纯数学补充数据，适合带最终校验目标的多轮推理/工具使用场景（用于RLVR） * `dpo.jsonl` --RLHF阶段偏好训练数据（DPO） * `pretrain_t2t_mini`✨ --`minimind-3` 轻量预训练数据，适合快速复现（推荐设置`max_seq_len≈768`） * `pretrain_t2t` --`minimind-3` 主线预训练数据（推荐设置`max_seq_len≈380`） * `rlaif.jsonl`✨ --RLAIF训练数据集，用于PPO/GRPO/CISPO等强化学习算法训练 * `sft_t2t_mini.jsonl`✨ --`minimind-3` 轻量SFT数据（用于快速训练Zero模型），推荐设置`max_seq_len≈768`，其中已混入一部分 Tool Call 样本 * `sft_t2t.jsonl` --`minimind-3` 主线SFT数据，适合完整复现，其中同样已混入 Tool Call 样本 训练参数 `max_seq_len` 目前指的是 tokens 长度，而非绝对字符数。 本项目tokenizer在中文文本上大约`1.5~1.7 字符/token`，纯英文的压缩比在`4~5 字符/token`，不同数据分布会有波动。 数据集命名标注的“最大长度”均为字符数，100长度的字符串可粗略换算成`100/1.5≈67`的tokens长度。 例如： * 中文：`白日依山尽`5个字符可能被拆分为[`白日`,`依`,`山`,`尽`] 4个tokens； * 英文：`The sun sets in the west`24个字符可能被拆分为[`The `,`sun `,`sets `,`in `,`the`,`west`] 6个tokens “推荐设置”给出了各个数据集上最大tokens长度的粗略估计。 须知 `max_seq_len` 可以激进 / 保守 / 均衡地调整，因为更大或更小均无法避免副作用：一些样本短于 `max_seq_len` 后被 padding 浪费算力，一些样本长于 `max_seq_len` 后被截断语义。 在算力效率与语义完整性之间找到平衡点即可 > MiniMind 主线训练数据组成与推荐组合示意图 说明 & 推荐训练方案 * `minimind-3` 主线推荐采用 `pretrain_t2t` + `sft_t2t` + `rlaif/agent_rl` 的阶段式训练组合。 * 想要最快速度从0实现Zero模型，推荐使用`pretrain_t2t_mini.jsonl` + `sft_t2t_mini.jsonl` 的数据组合 * 推荐具备一定算力资源或更在意效果的朋友完整复现 `minimind-3`；仅有单卡GPU或更在意快速复现的朋友强烈推荐 mini 组合。 * 当前 `sft_t2t / sft_t2t_mini` 已经混入 Tool Call 数据，因此通常不需要再额外做一轮独立的 Tool Calling 监督微调。 # 📌 模型 ## 结构 `minimind-3` Dense 使用 Transformer Decoder-Only 结构，整体配置已经向 `Qwen3` 生态对齐，方便后续转换到 `transformers / llama.cpp / ollama / vllm`： * 采用预标准化（Pre-Norm）+ RMSNorm。 * 使用 SwiGLU 激活函数。 * 使用 RoPE 旋转位置编码，并支持 YaRN 外推。 * `q_heads=8`、`kv_heads=4`，`max_position_embeddings=32768`，`rope_theta=1e6`。 `minimind-3-moe` 在相同结构上扩展 MoE 前馈层，实现上兼容 `Qwen3-MoE` 风格配置（去除 shared expert）。 * 当前默认配置为 `4 experts / top-1 routing`，用于以更低激活参数获得更高容量。 * Experts 继续增加后，实际耗时往往比同尺寸规模的 dense 模型高非常多，这和 “MoE 推理更快” 放在一起看会有点反直觉，但训练时 token 先按专家分桶、再分别做 forward，原生训练时带来的 `kernel` 启停和调度开销会急剧变重，这本身是很自然的事情。得靠支持 MoE kernel-fused 的算子库来优化，比如基于 `Triton` 的自定义 kernel、`DeepSpeed-MoE`、`Megatron-LM` 等等。当然，这个项目还是希望保留原生 PyTorch 的普适性，所以这里做的是现实的折中，在当前实现下，`4 experts / top-1` 这个甜点配置大约只比 dense 模型慢 `50%` 左右。 `minimind-3` 系列结构如下图： 修改模型配置见[./model/model_minimind.py](./model/model_minimind.py)，参考模型参数版本见下表： | Model Name | params | len_vocab | max_pos | rope_theta | n_layers | d_model | kv_heads | q_heads | note | |------------|--------|-----------|---------|------------|----------|---------|----------|---------|------| | minimind-3 | 64M | 6400 | 32768 | 1e6 | 8 | 768 | 4 | 8 | Dense | | minimind-3-moe | 198M-A64M | 6400 | 32768 | 1e6 | 8 | 768 | 4 | 8 | 4 experts / top-1 | | minimind2-small | 26M | 6400 | 32768 | 1e6 | 8 | 512 | 2 | 8 | 历史版本 | | minimind2-moe | 145M | 6400 | 32768 | 1e6 | 8 | 640 | 2 | 8 | 历史版本 | | minimind2 | 104M | 6400 | 32768 | 1e6 | 16 | 768 | 2 | 8 | 历史版本 | ## 模型配置 关于 LLM 的参数配置，[MobileLLM](https://arxiv.org/pdf/2402.14905) 对小模型做过一组很有代表性的系统研究。对 MiniMind 这类百M级模型而言，`d_model` 与 `n_layers` 的取舍不只是参数分配问题，也会直接影响训练稳定性与最终效果。 当前 `minimind-3` 主线选择 `dim=768, n_layers=8`，本质上是一种工程取舍：更浅的网络训练更快，同时 `dim` 也不至于过小而导致模式崩溃，因此能在训练效率、稳定性与最终效果之间取得相对均衡。 查看详细说明 Scaling Law 在小模型上往往会呈现出一些不同于大模型的现象。决定 Transformer 参数规模变化的核心参数，通常主要就是 `d_model` 和 `n_layers`： * `d_model`↑ + `n_layers`↓ -> 矮胖子 * `d_model`↓ + `n_layers`↑ -> 瘦高个 经典 Scaling Law 更强调训练数据量、参数量和训练步数的决定性作用，通常会弱化架构差异本身的影响；但在小模型区间，这个结论并不总是完全成立。 MobileLLM 的一个核心观察是：在参数量固定时，深度往往比宽度更重要。也就是说，相比“宽而浅”的结构，“深而窄”的模型更容易学到抽象概念。 例如当模型参数量固定在 `125M` 或 `350M` 时，`30~42` 层的狭长结构通常会优于 `12` 层左右的矮胖结构，在常识推理、问答、阅读理解等多个基准上都呈现出相近趋势。 这和 MiniMind 在训练过程中围绕 `d_model` 与 `n_layers` 做参数分配实验时观察到的现象是一致的。不过“深而窄”里的“窄”也有下限：当 `d_model < 512` 时，词嵌入维度过窄带来的劣势会明显放大，额外增加 layers 往往不足以完全弥补固定 `q_head` 下 `d_head` 偏小的问题。 相对地，当 `d_model > 1536` 时，继续增加层数往往比单纯继续加宽更划算，更容易带来更高的参数-效果收益。 作为参考，GPT-3 的参数设定如下： --- # 📌 实验 ## Ⅰ 训练开销 - **时间单位**：小时（h） - **成本单位**：人民币（￥）；`7￥ ≈ 1 美元` - **3090 租卡单价**：约 `1.3￥/h`（实际价格可自行参考） - **说明**：以下结果为 `minimind` 模型在单卡 `3090` 上的经验估算值，用于快速感知训练门槛 | Model Name | params | pretrain_t2t_mini | sft_t2t_mini | toolcall | RLAIF | |------------|--------|-------------------|--------------|----------|-------| | minimind-3 | 64M | ≈1.21h≈1.57￥ | ≈1.10h≈1.43￥ | ≈0.9h≈1.17￥ | ≈1.1h≈1.43￥ | | minimind-3-moe | 198M-A64M | ≈1.69h≈2.20￥ | ≈1.54h≈2.00￥ | ≈1.26h≈1.64￥ | ≈1.54h≈2.00￥ | --- 训练开销总结&预测 > `minimind-3` >> `pretrain_t2t_mini` + `sft_t2t_mini` 单卡 `3090`，`1 epoch` 预计约 `2.31` 小时，成本约 `3.0` 元人民币 可从 0 训练出 `minimind-3 Zero` 对话模型。 > `minimind-3-moe` >> `pretrain_t2t_mini` + `sft_t2t_mini` 单卡 `3090`，`1 epoch` 预计约 `3.23` 小时，成本约 `4.2` 元人民币 可快速得到 `minimind-3-moe` 的基础对话版本。 > 以上均为估算值，仅用于快速感知训练门槛。 基于单卡 `NVIDIA 3090`，`minimind zero` 从 0 训练依然可以控制在约 `2` 小时量级，个人开发者也能较低门槛地快速上手。 若采用更高规格的多卡环境，例如 `8x H100`，总训练时长还可进一步压缩至分钟级。以尽可能低的门槛实现可复现、可上手、可持续迭代的 LLM 训练体验，这也是 MiniMind 系列一直希望坚持的方向。低成本快速复现并不是噱头，下面保留一个早期的 Zero 风格样例对话供参考： ```text 👶: 请介绍一下自己。 🤖️: 作为人工智能，我没有实际的生活，也没有自我意识，所以没有自己的生活。我被设计成能够帮助用户解答问题、提供信息、进行对话等。我的设计和功能是由计算机科学、机器学习、人工智能技术和算法所构建的，我通过编程和算法实现。 👶: 推荐一些杭州的特色美食吧。 🤖️: 杭州有很多美味的特色美食，比如鸡肉串、鳗鱼头、水煮鱼、豆腐脑等。这些美食都非常美味，有着独特的风味和口感，绝对是值得一试的美食。 👶: 请为我讲解“大语言模型”这个概念。 🤖️: 大语言模型是一种人工智能技术，它使用大量的文本数据来训练模型，然后在不断的迭代中不断优化模型。它的目标是根据输入的数据来生成符合特定需求的文本，这些文本可以是任何你感兴趣的主题。 👶: Introduce the history of the United States, please. 🤖️: 您提到的“Introok's the believeations of theument." 这个名字来源于中国古代的"groty of of the change." ``` 尽管该版本已经具备基础对话能力，但事实知识与泛化效果仍较有限；它更适合作为 Zero 训练路线可行性的早期参考。 Zero 模型权重保存为 `full_sft_zero_768.pth`（见下文 MiniMind 模型文件链接），如有兴趣可下载体验其对话效果。 --- ## Ⅱ 主要训练（必须） > 所有训练脚本均 `cd ./trainer` 目录执行 ### 1' 预训练 (Pretrain): LLM 首先要学会的是先把尽可能多的基础知识和语言规律吸收到参数里。只有这一步打稳了，模型后面才有能力去理解问题、组织表达，并逐步形成像样的生成能力。预训练做的事情，本质上就是让模型先埋头读大量文本，例如 Wiki 百科、新闻、书籍、对话语料等，从中学习事实知识、语言模式以及上下文之间的统计关系。这个阶段通常是“无监督”的：人类不需要逐条告诉模型哪里对、哪里错，而是让它自己从海量文本里总结规律，逐步建立起对世界知识和语言结构的内部表征。 更直白地说，模型在这一阶段的核心目标就是**学会高质量地词语接龙**。例如输入“秦始皇”，它要能够继续生成“是中国历史上的第一位皇帝”这类符合语义与常识的后续内容。 ```bash # 方式1 torchrun --nproc_per_node 1 train_pretrain.py # 1即为单卡训练，可根据硬件情况自行调整 (设置>=2) # 方式2 python train_pretrain.py ``` > 训练后的模型权重文件默认每隔`save_interval步`保存为:`pretrain_*.pth`（*为模型具体dimension，每次保存时新文件会覆盖旧文件） > `768dim` 配置在预训练阶段的 loss 曲线 ```bash # 可对预训练结果做简单测试： python eval_llm.py --weight pretrain 💬: 为什么天空是蓝色的 🧠: 天空之所以看起来是蓝色的，主要是因为太阳光进入大气层后，短波长的蓝光更容易被空气分子散射，因此人眼从各个方向接收到的蓝光会更多。 💬: 解释什么是机器学习 🧠: 机器学习是人工智能的一个重要分支，它通过数据训练模型，使系统能够自动学习规律，并在分类、预测、推荐、自然语言处理等任务中持续改进效果。 ``` ### 2' 有监督微调 (Supervised Fine-Tuning): SFT 并不只是把模型调成“更会聊天”，它同样可以继续向模型中灌入新的知识、行为模式和回答风格。尤其是像 MiniMind 当前主线这样体量达到 `14GB` 的 SFT 数据，本身就已经不只是简单的格式对齐，而更接近一种带有 `mid training` 性质的持续强化过程。 如果把预训练理解为先让模型广泛地读书、积累基础语言能力，那么 SFT 更像是在高质量、更有目标的数据上继续深加工。一方面，它会让模型适应多轮对话、问答、工具调用和思考标签等交互形式；另一方面，它也会继续把特定知识分布、任务模式和助手风格压进参数里。 具体到 MiniMind 里，SFT 阶段会让模型适应当前仓库使用的多轮对话模板。模型会逐渐理解 `user / assistant / system / tool` 等角色结构，同时进一步强化指令跟随、稳定回复和任务完成能力。 当前训练时会对指令和回答长度做截断控制，主要是为了兼顾显存占用与训练效率；如果后续需要更长上下文，只需要继续准备少量长样本做增量微调即可。在推理时通过启用 YaRN 外推，可以免训练地将上下文长度扩展到 2048 及以上。 ```bash # 方式1 torchrun --nproc_per_node 1 train_full_sft.py # 方式2 python train_full_sft.py ``` > 训练后的模型权重文件默认每隔`save_interval步`保存为: `full_sft_*.pth`（*为模型具体dimension，每次保存时新文件会覆盖旧文件） > `768dim` 配置在 SFT 阶段的 loss 曲线 ```bash # 可对SFT结果做简单测试： python eval_llm.py --weight full_sft 💬: 解释什么是机器学习 🧠: 机器学习是人工智能的核心技术之一，通过算法让计算机从数据中学习规律，并持续改进预测或决策效果，常见应用包括推荐系统、图像识别、语音识别和自然语言处理。 💬: 推荐一些中国的美食 🧠: 例如北京烤鸭、兰州拉面、四川火锅、广东早茶、小笼包和麻婆豆腐等，这些美食分别代表了不同地区的风味特点，也很适合作为了解中国饮食文化的入门选择。 ``` ## Ⅲ 其它训练（可选） > 所有训练脚本均 `cd ./trainer` 目录执行 ### 3' 知识蒸馏 (Knowledge Distillation, KD) 知识蒸馏大体可以分成黑盒和白盒两类，MiniMind 当前主线两种思路都有涉及，只是侧重点不同。 * 黑盒蒸馏：更常见，也更贴近当前主线的实际做法。严格来说，它本质上仍然是面向教师输出结果的监督微调，也就是基于硬标签继续训练；只是随着 LLM 的流行，这类“对着强模型输出做 FT”的做法也逐渐被广义地归入了蒸馏范畴，因此通常被称为黑盒蒸馏。它重点学习的是答案、风格和行为模式，学生模型只能看到“老师说了什么”，却看不到老师内部是如何做出这个判断的。像 `DeepSeek R1`、`Qwen3` 的高质量回答，以及 `tool call`、`reasoning`、思维链等数据，都可以看作黑盒蒸馏信号；MiniMind 当前主线 `full_sft` 数据里，其实已经混入了相当一部分这样的思路。 * 白盒蒸馏：更进一步，不只学习教师给出的最终输出，还去学习教师在 token 分布层面的偏好。相比黑盒蒸馏，它额外利用了教师模型输出层更细粒度的分布信息，因此学生模型学到的不只是“标准答案”，还包括教师在候选 token 之间的相对倾向。对应到 `train_distillation.py`，当前实现是在已经完成 SFT 的权重基础上，继续用教师模型提供的分布信号来训练学生模型，因此更适合作为理解 MiniMind 蒸馏流程的参考实现。 黑盒蒸馏本质上等价于对 teacher 生成答案做监督微调： ```math \mathcal{L}_{blackbox} = \mathrm{CE}(y_{teacher}, p_{student}) ``` 白盒蒸馏则通常在监督损失之外，再额外拟合教师分布： ```math \mathcal{L}_{whitebox} = \alpha \mathcal{L}_{CE} + (1-\alpha) T^2 \mathrm{KL}(p_t^T \parallel p_s^T) ``` 仓库中提供的 `train_distillation.py` 更适合作为理解白盒蒸馏流程的参考实现：它完整展示了教师/学生双模型加载、`CE + KL` 混合损失、温度缩放、MoE 与 dense 组合蒸馏，以及断点续训和分布式训练等关键细节。 ```bash # 方式1 torchrun --nproc_per_node 1 train_distillation.py # 方式2 python train_distillation.py ``` ### 4' LoRA (Low-Rank Adaptation) LoRA 是一种常见的参数高效微调（Parameter-Efficient Fine-Tuning, PEFT）方法。相比全参数微调，它只更新少量新增参数，而保留原始模型主体权重不变，因此训练成本更低，也更适合做垂直场景适配。 它的核心思想是在原有权重矩阵旁引入低秩增量分支，仅训练这部分低秩参数，从而用较小代价完成能力迁移。相关实现可见 `model_lora.py` 和 `train_lora.py`，整个流程均为纯手写实现，不依赖第三方封装。 ```bash # train_lora.py 在 CPU 上通常也能比较轻快地完成 # 方式1 cd trainer && torchrun --nproc_per_node 1 train_lora.py # 方式2 cd trainer && python train_lora.py ``` > 训练后的模型权重文件默认每隔`save_interval步`保存为: `lora_xxx_*.pth`（*为模型具体dimension，每次保存时新文件会覆盖旧文件） LoRA 很适合处理“如何在尽量保留通用能力的前提下，让模型快速适应私有领域或垂直场景”这类问题。例如基础模型医学知识不足时，就可以在原有模型之上叠加一层面向医疗场景的 LoRA 权重，以较小代价获得更好的领域表现。 通常只需要准备同样的多轮对话格式数据，放置到 `lora_xxx.jsonl`，再从仓库根目录执行 `cd trainer && python train_lora.py`，即可得到新的 `LoRA` 模型权重。 例1：垂域数据 ```jsonl {"conversations": [{"role": "user", "content": "请问颈椎病的人枕头多高才最好？"}, {"role": "assistant", "content": "颈椎病患者选择枕头的高度应该根据..."}]} {"conversations": [{"role": "user", "content": "请问xxx"}, {"role": "assistant", "content": "xxx..."}]} ``` 例2：自我认知数据 ```jsonl {"conversations": [{"role": "user", "content": "你叫什么名字？"}, {"role": "assistant", "content": "您好，我名叫 MiniMind，是由 Jingyao Gong 开发的人工智能助手。我可以帮助你解决各种问题，包括翻译、推荐内容等等。"}]} {"conversations": [{"role": "user", "content": "你是谁"}, {"role": "assistant", "content": "您好，我是 MiniMind，由 Jingyao Gong 创建。我的主要职责是提供准确的回答和帮助。"}]} ``` 此时可将【基模 + LoRA 权重】组合使用，从而在保留基础模型通用能力的同时，增强特定领域表现。训练完成后也可以通过 `eval_llm.py` 做简单测试。 ```bash # 注意：weight参数指定基础模型类型，需与train_lora训练时使用的基础模型保持一致 python eval_llm.py --weight full_sft --lora_weight lora_medical 👶: 我最近经常感到头晕，可能是什么原因？ 🤖️: 头晕可能是由于多种原因，包括但不限于：心脏疾病、眼睛损伤、过敏反应、皮肤问题、过敏反应、压力、焦虑、疲劳、药物副作用、性功能障碍或者其他... python eval_llm.py --weight full_sft --lora_weight lora_identity 👶: 你和openai是什么关系？ 🤖️: 我是 MiniMind，一个由 Jingyao Gong 开发的人工智能助手。我通过自然语言处理和算法训练来与用户进行交互。 ``` PS：如果有更充足的数据，也可以直接做 `full_sft` 全参微调；不过这通常需要更谨慎地混合通用数据与领域数据，否则很容易因为过拟合垂域样本而损失模型原有的通用性。 > `LoRA` 权重可合并回基础模型并导出为新的完整模型权重，可使用 `scripts/convert_model.py` 中的 `convert_merge_base_lora`： ```bash cd scripts && python convert_model.py ``` ### **5' 工具调用 & 自适应思考** `2026-03` 起，仓库移除了独立的 `train_reason.py`。 当前版本不再单独维护 `reason_*.pth` 权重，而是统一通过 `chat_template`、`` 标签、`open_thinking` 开关以及后续 SFT / RLAIF 流程来建模“是否显式输出思考过程”。 #### 5.1 Tool Calling 当前 `toolcall` 能力已经并入 `sft_t2t` / `sft_t2t_mini` 主线数据，因此通常不再需要额外单独训练一轮 Tool Calling；默认的 `full_sft` 权重已经具备基础 Tool Call 能力。当前这部分训练数据主要由 `qwen3-4b` 采样约 `10w` 条构成，工具列表也主要覆盖约 `10` 个模拟的自定义工具（例如查询时间、数学计算、获取天气等），因此目前还谈不上明确的泛化能力。其中 Tool Calling 样本统一沿用了 OpenAI 风格的多轮消息格式： ```jsonl { "conversations": [ {"role": "system", "content": "# Tools ...", "tools": "[...]"}, {"role": "user", "content": "帮我算一下 256 乘以 37 等于多少"}, {"role": "assistant", "content": "", "tool_calls": "[{\"name\":\"calculate_math\",\"arguments\":{\"expression\":\"256 * 37\"}}]"}, {"role": "tool", "content": "{\"result\":\"9472\"}"}, {"role": "assistant", "content": "256 乘以 37 等于 9472。"} ] } ``` 其中 `tools` 挂在 `system` 消息上，`tool_calls` 挂在 `assistant` 消息上；训练时再由 `chat_template` 自动展开为 `...` 与 `...` 片段，因此现在可以直接学习原生 tool call 格式。 Tool Calling 的 chat template 已统一支持解析为： ```text {"name": "...", "arguments": {...}} {...tool result...} ``` 也可以直接通过 `eval_toolcall.py` 做简单测试： ```bash python eval_toolcall.py --weight full_sft 💬: 现在几点了？ 🧠: {"name": "get_current_time", "arguments": {"timezone": "Asia/Shanghai"}} 📞 [Tool Calling]: get_current_time ✅ [Tool Called]: {"datetime": "2026-03-15 17:18:22", "timezone": "Asia/Shanghai"} 🧠: 现在是2026年3月15日17时18分22秒。 ``` #### 5.2 Adaptive Thinking `minimind` 将显式思考能力统一到了模板层，这也和当前很多主流大模型的模板设计保持一致： - `open_thinking=0`：默认注入空的 `\n\n`，模型更倾向于直接回答； - `open_thinking=1`：模板会预先注入 `` 起始标签，模型再继续输出显式思考过程与最终回答； - CLI、OpenAI-API、WebUI 三个入口均支持该开关。 更准确地说，目前不再“单独训一个思考模型”，而是把“是否显式思考”下沉到 `chat_template`。模板层会先预留 `` 这一结构，同一个模型在推理时再通过 `open_thinking` 动态切换；在训练时，则通过混合空 `think`、显式 `reasoning_content` 与 `thinking_ratio` 采样，让模型逐步见过“该想时想、该直答时直答”的混合模式。 ```bash # 测试回答 python eval_llm.py --load_from ./minimind-3 --open_thinking 1 ``` OpenAI-API-SDK 用法： ```python response = client.chat.completions.create( model="minimind", messages=[{"role": "user", "content": "你是谁？"}], # ... extra_body={"chat_template_kwargs": {"open_thinking": True}} # 思考开关 ) ``` 注：当前同时开启 Tool Call 与显式思考时，模型通常并不太会稳定地输出思考过程。原因在于现阶段训练数据里还缺少“reasoning 与 tool call 同时存在”的联合蒸馏样本，因此模型尚未充分学会这两种能力的协同表达。 ## Ⅳ 强化学习（可选） 在 LLM 的后训练实践中，常见的强化学习路径主要有两条： 1. **基于人类反馈的强化学习 (Reinforcement Learning from Human Feedback, RLHF)** - 通过**人类**对模型输出的偏好进行评价来训练模型，使其生成更符合人类价值观和偏好的内容。 2. **基于AI反馈的强化学习 (Reinforcement Learning from AI Feedback, RLAIF)** - 使用**AI模型**或其他可自动验证的机制来提供反馈，而不直接依赖人类标注。 - 这里的“AI feedback”在广义上也可以扩展到规则奖励、Ground Truth 校验、代码解释器、环境反馈等自动化信号。 | 类型 | 裁判 | 优点 | 缺点 | |-------|----|-----------|------------| | RLHF | 人类 | 更贴近真实人类偏好 | 成本高、效率低 | | RLAIF | 模型 | 自动化、可扩展性强 | 可能偏离人类真实偏好 | 二者本质上都属于利用某种形式的"**反馈**"来优化模型行为的强化学习范式。 不过在具体实践里，它们并不只是反馈来源不同：奖励是否可验证、是否连续、是否依赖环境交互、是否延迟到整轮结算，都会直接影响训练形态与工程实现。 ### 👀 PO算法的统一视角 在介绍实现具体算法之前，我先以个人理解的极简视角，阐述所有Policy Optimization (PO)算法的统一共性。 所有RL算法的本质都只是在优化一个期望： $$\mathcal{J}_{PO} = \mathbb{E}_{q \sim P(Q), o \sim \pi(O|q)} \left[ \underbrace{f(r_t)}_{\text{策略项}} \cdot \underbrace{g(A_t)}_{\text{优势项}} - \underbrace{h(\text{KL}_t)}_{\text{正则项}} \right]$$ 训练时，只需**最小化负目标函数**，即: $\mathcal{L}_{PO} = -\mathcal{J}_{PO}$ 这个框架只包含三个核心组件： * **策略项** $f(r_t)$: 如何使用概率比 $r_t$? 即告诉模型新旧策略偏差有多大，是否探索到了更好的token * **优势项** $g(A_t)$: 如何计算优势 $A_t$, 这很重要！大模型算对定积分也不足为奇，小模型回答对加减法优势通常都是正的 * **正则项** $h(\text{KL}_t)$: 如何约束变化幅度 $\text{KL}_t$, 既防止跑偏又防止管的太死 （展开）符号说明 | 符号 | 含义 | 说明 | 值域 | |------|------|------|------| | $q$ | 问题/提示词 | 从数据集 $P(Q)$ 中采样 | - | | $o$ | 模型输出序列 | 由策略 $\pi$ 生成 | - | | $r_t$ | 概率比 | $r_t = \frac{\pi_\theta(o_t \mid q, o_{ 训练后的模型权重文件默认每隔`save_interval步`保存为: `dpo_*.pth`（*为模型具体dimension，每次保存时新文件会覆盖旧文件） ### 7' 基于 AI 反馈的强化学习 (Reinforcement Learning from AI Feedback, RLAIF) 稍微花篇幅解释一下，我还是更想把这一节叫作 `RLAIF`，虽然严格来说，这个命名并不完全准确。像 RLVR 这类依赖可验证奖励的路线，本身有相对独立的脉络，很难被简单并进狭义的 AI feedback 里。 但如果把“AI”理解得稍微宽一点，我又觉得这个名字并非完全说不通：奖励既可以来自奖励模型、judge model 这类显式的智能体，也可以来自规则函数、Ground Truth校验、工具调用结果、环境返回状态这类可自动获得的信号。规则足够复杂、符号系统足够丰富时，它们和“智能反馈”之间的边界，本来就未必那么泾渭分明。 因此这一章更想讨论的，其实是 LLM 在 SFT 之后，借助各种**非人工、可自动获得的反馈信号**继续做强化学习优化的方法。比如数学题答案是否正确、工具调用执行代码能否通过测试用例、推理过程是否符合格式...都可以自动化判断。 对于单轮可验证任务，这类反馈往往更接近“即时打分”；而在 Agentic RL 场景中，奖励则更常表现为多步交互后的延迟结算，甚至直接来自环境本身。 它们共同的特点通常都是**On-Policy**与**可扩展性强**——不需要昂贵的人工标注，可以生成海量训练样本，让模型在在线大量试错中快速进化。 MiniMind 着手实现**2+N**种基本+前沿的RLAIF方法： * **PPO**、**GRPO** 被大规模验证的经典RL算法 * N种前沿RL算法（不定期以Exp性质更新） **1️⃣ 数据集准备 (必须)** 当前主线使用 `rlaif.jsonl` 作为 RLAIF 训练数据，体量约 `20MB`，比早期 `rlaif-mini.jsonl` 更完整，更适合直接验证 PPO / GRPO / CISPO 的训练效果。 数据格式与 SFT 一致，但 assistant 字段不需要真实内容，因为训练过程中完全由 $\Pi$ 策略模型实时采样生成。因此形如： ```json { "conversations": [ {"role": "user", "content": "请解释一下什么是光合作用？"}, {"role": "assistant", "content": "无"} ] } ``` RLAIF的训练过程中，模型会基于user的问题生成1或多个候选回答，然后由奖励函数/模型对回答打分， 分数高的回答会被鼓励（增加 $\Pi$ 策略概率），分数低的回答会被抑制（降低 $\Pi$ 策略概率）。这个"打分->调整"的循环就是强化学习的核心。 **2️⃣ 奖励机制准备 (必须)** RLAIF训练需要某种可计算的奖励信号；它可以来自奖励模型，也可以来自规则函数、Ground Truth 校验或环境反馈。MiniMind 当前默认演示的是 Reward Model 路线。 此处选取小型且高质量的 `InternLM2-1.8B-Reward` ([ModelScope](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm2-1_8b-reward) | [HuggingFace](https://huggingface.co/internlm/internlm2-1_8b-reward)) 作为基础奖励模型。 下载奖励模型后需要放置在minimind项目的**同级目录**下，推荐结构如下： ``` root/ ├── minimind/ # MiniMind项目 │ ├── model/ │ └── ... └── internlm2-1_8b-reward/ # 奖励模型 ├── config.json ├── model.safetensors └── ... ``` **奖励机制选择与MiniMind限制说明（点击展开）** **1. 奖励机制的多样性** RLAIF中的"奖励信号"来源可以非常灵活： - **Model-based奖励**：可使用专门的Reward Model（如InternLM2-Reward），也可使用通用LLM+提示词进行打分（如Qwen3-as-a-Judge）。奖励模型规模和架构均可自由选择。 - **Rule-based奖励**：可以基于规则函数构造奖励信号，例如： - 数学题答案正确性验证（Ground Truth对比） - SQL执行成功率与结果准确性 - 代码解释器运行结果（pass@k） - 工具调用返回状态（API成功/失败） - 格式合规性检查（JSON/XML解析） - 推理链完整性评估（CoT步骤数） - **Environment-based奖励**：在Agent场景中，环境反馈本身即为天然奖励（如游戏得分、Research完整度、任务完成度）。 任何能够量化"回答质量"的机制都可作为RL的奖励来源。DeepSeek R1就是典型案例：使用规则函数验证数学答案正确性作为奖励，无需额外的Reward Model。 **2. MiniMind限制：奖励稀疏问题** RLAIF训练既可以针对推理模型也可以针对非推理模型，区别仅在于格式。 然而对于 MiniMind 这种 0.1B 参数量、能力较弱的模型，在通用任务（如 R1 风格的数学数据集）上会遇到严重的奖励稀疏（Reward Sparsity）问题： - **现象**：模型生成的候选回答几乎全部错误，导致所有奖励分数 $r(x,y) \approx 0$ - **后果**：优势函数 $A(x,y) = r(x,y) - b(x) \approx 0$，策略梯度信号消失，无法有效更新参数 $\theta$ 如同让小学生做高考数学题，无论尝试多少次都得零分，无法通过分数差异学习改进策略。这属于 RL 算法在奖励稀疏场景下的根本限制。 为缓解此问题，MiniMind的实现选择了**model-based的连续性奖励信号**： - Reward Model输出连续分数（如-2.5到+3.0），而非二元的0/1 - 即使回答质量都差，也仍能区分“更差”(-3.0)和“没那么差”(-2.8)的细微差异。所以这种**稠密且连续**的奖励信号能够为优势函数 $A(x,y)$ 提供非零梯度，使得策略网络得以渐进式优化 - 也可以混合多种奖励源: $r_{\text{total}} = \alpha \cdot r_{\text{model}} + \beta \cdot r_{\text{rule}}$ (例如既可以检测 thinking 标签格式奖励，又可以综合回答本身质量的 reward 分数) - MiniMind 实践中避免直接使用 rule-based 二元奖励 + 超纲难度数据（如 MATH500），易导致奖励全零； - 监控训练时观察奖励分数的方差 $\text{Var}(r)$，若持续接近0则需调整数据或奖励机制 **对于生产级大模型的Agentic RL场景**： 在真实Agent系统（代码生成、工具调用、检索-规划-执行的多轮链路）中，奖励是“延迟整轮结算”的不同范式： - LLM需要逐token生成工具调用指令（tool_call），经历解析（tool_parse）、工具执行（tool_exec），再把结果拼接回上下文继续下一步；循环往复直到完成。 - 一次完整的任务链路包含多次调用+思考，直到终止条件满足时计算一次总reward（如任务是否完成、测试是否通过、目标是否命中）。 因此，Agentic RL更接近稀疏/延迟奖励设定：梯度回传在“整轮结束后”才发生，和非Agentic RL任务在对话单轮上“即时评分即时更新”有很大不同。 这也解释了Agent任务上更偏向环境反馈（environment-based reward），而非凭Reward Model进行静态打分。 - **环境交互反馈**：最终以执行结果为准（代码是否跑通、API是否返回成功、子目标是否完成）； - **Model-based奖励局限**：对长链路、可执行语义的全貌捕捉有限，且大概率和真实环境反馈不一致（reward hacking）。 --- #### 7.1 [Proximal Policy Optimization](https://arxiv.org/abs/1707.06347) PPO 是 2017 年 OpenAI 提出的非常经典的强化学习算法，也是 LLM RL 领域最常见的基线方法之一。 **PPO损失**： $$\mathcal{L}_{PPO} = -\mathbb{E}\left[\min(r_t \cdot A_t, \text{clip}(r_t, 1-\varepsilon, 1+\varepsilon) \cdot A_t)\right] + \beta \cdot \mathbb{E}[\text{KL}]$$ 其中： - **策略项**: $f(r_t) = \min(r_t, \text{clip}(r_t, 1-\varepsilon, 1+\varepsilon))$ (裁剪概率比防止更新过激) - **优势项**: $g(A_t) = R - V(s)$ (通过Critic网络估计价值函数) - **正则项**: $h(\text{KL}_t) = \beta \cdot \mathbb{E}[\text{KL}]$ (全局KL散度约束) 对比DPO而言， - DPO (Off-Policy)：训练数据是静态偏好对（chosen vs rejected），可以反复使用同一批数据训练多个 epoch，像传统监督学习一样。数据效率高、成本低，且无需 Reward Model。 - PPO (On-Policy)：必须用当前策略实时采样新数据，旧策略数据只能有限复用，否则就会出现 distribution shift。虽然 importance sampling 和 clip 允许轻微偏移，但本质上仍要求数据来自较新的策略。数据效率更低，但更适合探索式学习。 简单来说： - 前者按离线预定的「好/坏标准」学习； - 后者则基于最新 policy 在线采样并实时纠偏。 MiniMind 的 PPO 实现包含 Actor（生成回答）、Critic（评估回答价值）以及完整的 GAE（Generalized Advantage Estimation）优势函数计算。 **训练方式**： ```bash # 方式1 torchrun --nproc_per_node N train_ppo.py # 方式2 python train_ppo.py ``` > 训练后的模型权重文件默认每隔`save_interval步`保存为: `ppo_actor_*.pth`（*为模型具体dimension） > MiniMind 在 PPO 训练阶段的优化走势 从训练曲线可以看出，PPO存在**reward提升缓慢**的问题。私以为这主要源于**PPO双网络联合优化**方法：Critic需要逐步收敛以准确估计价值函数，而Actor的策略更新依赖Critic提供的优势估计，两者相互依赖形成复杂的优化过程。训练初期Critic估计不准会影响Actor梯度方向，导致整体收敛缓慢。此外，PPO 需要同时维护两个网络，在当前实现下显存占用约为单网络方法的 1.5–2 倍。 #### 7.2 [Group Relative Policy Optimization](https://arxiv.org/pdf/2402.03300) 2025 年初，随着 DeepSeek-R1 火爆出圈，来自 DeepSeekMath 论文的 GRPO 也迅速进入主流视野，一度成为最受关注的 RL 算法之一。不过 AI 领域向来迭代极快。时至今日，GRPO 更多已经演变成各类 XXPO 变体（如 DAPO、GSPO、CISPO 等）的共同基线。一句话概括它的核心创新，就是“分组相对价值估计”。 **GRPO损失**： $$\mathcal{L}_{GRPO} = -\mathbb{E}\left[\min(r_t \cdot A_t, \mathrm{clip}(r_t, 1-\varepsilon, 1+\varepsilon) \cdot A_t) - \beta \cdot \text{KL}_t\right]$$ 其中： - **策略项**: $f(r_t) = \min(r_t, \mathrm{clip}(r_t, 1-\varepsilon, 1+\varepsilon))$ (使用概率比的对称 clip 裁剪) - **优势项**: $g(A_t) = \frac{R - \mu_{group}}{\sigma_{group}}$ (组内归一化，消除Critic网络) - **正则项**: $h(\text{KL}_t) = \beta \cdot \text{KL}_t$ (token级KL散度约束) 对于同一个问题，模型生成 N 个回答并计算各自奖励，再用组内平均奖励作为 baseline。高于 baseline 的回答被鼓励，低于 baseline 的回答被抑制，因此无需额外训练 critic 网络。 GRPO 更显著的问题是退化组（Degenerate Groups）：如果某个问题上 N 个回答的奖励几乎一样，那么这一组的学习信号就会接近 0。在 MiniMind 这种超小模型上，这个问题尤其明显，所以训练必须限制在合理的能力边界内。 **训练方式**： ```bash # 方式1 torchrun --nproc_per_node N train_grpo.py # 方式2 python train_grpo.py ``` > 训练后的模型权重文件默认每隔`save_interval步`保存为: `grpo_*.pth` > MiniMind 在 GRPO 训练阶段的优化走势 从训练曲线可以看出，GRPO的**reward呈现更加稳定的上升趋势**，达到4左右，说明GRPO本身能更好地利用RLAIF信号。Policy Loss整体下降平稳，相比PPO的双网络优化，GRPO单网络架构训练更稳定且收敛上限更高。 #### 7.3 [Clipped Importance Sampling Policy Optimization](https://huggingface.co/papers/2506.13585) 我自己在众多眼花缭乱的XXPO里反而对它印象很深，CISPO没有重新发明一整套复杂框架，而是抓住了 PPO/GRPO 里一个长期让人别扭的问题——ratio 被 clip 之后，梯度流直接就被硬截断了。 CISPO 的关注点并不是重新设计 group baseline，而是用非常小的 loss 改动，更直接地修正这个问题。 **CISPO损失**： $$\mathcal{L}_{CISPO} = -\mathbb{E}\left[\min(r_t, \varepsilon_{max}) \cdot A_t \cdot \log \pi_\theta(a_t|s) - \beta \cdot \text{KL}_t\right]$$ 其中： - **策略项**: $f(r_t) = \min(r_t, \varepsilon_{max}) \cdot \log \pi_\theta(a_t|s)$ (ratio 只作为裁剪后的权重) - **优势项**: $g(A_t) = \frac{R - \mu_{group}}{\sigma_{group}}$ (可直接沿用 GRPO 的组内相对优势) - **正则项**: $h(\text{KL}_t) = \beta \cdot \text{KL}_t$ (token级KL散度约束) CISPO在GRPO基础上，把原本容易被clip成常数的策略项改写成“裁剪权重 × log 概率”的形式。这样ratio即使被截断，也不会把梯度路径一起截断。因此可以直接把CISPO视作GRPO的loss变体来实现，而不是单独维护一套独立脚本。这里不再单列实验。只需在 `train_grpo.py` 把 `loss_type` 配置为 `cispo`，其余训练流程仍沿用 GRPO 的分组采样、奖励计算与优势构造逻辑即可。 #### 7.4 Agentic RL 🔥 “Agentic”的概念其实很大，所以这里说的 Agentic 只能是一个相对狭义的版本：它更聚焦于让 MiniMind 这样的~百M小模型在有限工具集上学会基础的调用、观察与再规划能力，而不是去覆盖完整 Agent 系统里更大范围的状态管理、长期记忆与复杂工作流编排。 `2026-03` 起，仓库新增 `train_agent`，开始支持一种更贴近真实交互流程的多轮 Tool-Use RL。这是我自己很喜欢的一个训练脚本：它把 RLVR / RLAIF 风格的数据组织方式与 online RL 的 rollout 过程揉在了一起，中间来回调过很多版，也踩过收敛失败、奖励 hack、多轮上下文错位之类的 bug，最后仍然保持了 MiniMind 一贯的简洁性和可读性。 此部分的数据为 `agent_rl.jsonl` / `agent_rl_math.jsonl`。它们相比普通对话数据多了 `gt` 作为最终校验目标；若把一条样本记作 $(x, \mathcal{T}, gt)$，那么训练时优化的对象就不再是单轮回答 $y$，而是一条多轮轨迹 $\tau$： $$ \tau = (a_1, o_1, a_2, o_2, \dots, a_T), \quad a_t \sim \pi_\theta(\cdot \mid s_t, \mathcal{T}) $$ 其中 `chat_template` 统一组织 `tools / tool_calls / tool` 消息；若某一步生成了 `tool_call`，就执行工具并把 observation 拼回上下文，再继续 rollout。 主线流程可以压缩成： $$ \texttt{rollout batch} \rightarrow \texttt{calculate rewards} \rightarrow \texttt{policy update} $$ reward 也是对整条轨迹联合打分： $$ R(\tau) = R_{\text{answer}} + R_{\text{tool}} + R_{\text{format}} + R_{\text{rm}} - R_{\text{unfinished}} $$ 这里同时考虑工具调用合法性、`gt` 命中、格式闭合、未完成惩罚与 Reward Model 分数。和普通 PPO / GRPO 相比，这里是多轮 rollout、延迟 reward。 **训练方式**： ```bash # ① 默认使用torch做rollout # 方式1 torchrun --nproc_per_node N train_agent.py # 方式2 python train_agent.py ``` ```bash # ② 使用sglang做rollout # 需先启动sglang server： python -m sglang.launch_server --model-path ./minimind-3 --attention-backend triton --host 0.0.0.0 --port 8998 # 训练参数可参考： python train_agent.py --rollout_engine sglang --sglang_base_url http://localhost:8998 --sglang_shared_path ./ckpt_mm --data_path ../dataset/agent_rl_math.jsonl --use_wandb ``` > 训练后的模型权重文件默认每隔`save_interval步`保存为: `agent_*.pth` > MiniMind 在 Agentic RL 训练阶段的优化走势 这里顺带提一下 `rollout_engine`。所谓“训推分离”，就是把 **参数更新** 和 **轨迹展开** 拆开：训练侧负责优化 policy，rollout 侧负责高吞吐采样，对上统一表现为“给我 prompt，我返回 rollout 结果；训练完以后，再把新权重同步回来”。因此训练脚本并不需要关心底层到底是本地 `generate` 还是远端 `inference` 引擎。需要说明的是，当前实现仍是**同步**模式（采样完一批再更新），还不是纯 rollout buffer 的异步训练。 > MiniMind 中训练侧、轨迹侧与 rollout 侧解耦的 RL 结构示意图 如果类比到更大规模系统里，它其实已经具备 openrlhf/verl/slime 等大规模RL框架的味道： - 左边是训练侧，负责 policy 更新 - 右边是 rollout / inference 侧，负责吞吐采样 - 中间通过轨迹与权重同步完成衔接 - 工具执行与环境反馈本身不直接进入 loss，但会直接影响整条轨迹的 reward 质量 所以我自己会把这套实现视为 MiniMind 里一个很有意思的过渡版本：虽然还远不是工业级 Agent 训练框架，但已经把 **模板组织、工具执行、多轮 rollout、延迟奖励、训推分离** 这些关键元素真正实现了最小串联（也许目前没有比它更简洁的了）。 ```bash # 测试最终模型 Tool Use 的能力 python eval_toolcall.py --weight agent 💬: 现在几点了？ 🧠: {"name": "get_current_time", "arguments": {"timezone": "Asia/Shanghai"}} 📞 [Tool Calling]: get_current_time ✅ [Tool Called]: {"datetime": "2026-03-15 21:22:33", "timezone": "Asia/Shanghai"} 🧠: 现在是2026年3月15日21时22分33秒（北京时间）。 💬: 帮我生成一个1到1000的随机数，然后计算它的平方 🧠: {"name": "random_number", "arguments": {"min": 1, "max": 1000}} 📞 [Tool Calling]: random_number ✅ [Tool Called]: {"result": 71} 🧠: {"name": "calculate_math", "arguments": {"expression": "71**2"}} 📞 [Tool Calling]: calculate_math ✅ [Tool Called]: {"result": "5041"} 🧠: 生成的1到1000的随机数是71，根据计算结果，71的平方等于5041。 ``` > 基于AgentRL训练结果测试，支持思考展示、工具选择与多轮 Tool Use 交互 ### 🖊️ RL小结 我们收束回“**统一框架**”：不同 PO 算法本质上只是对三个核心组件的不同实例化，见下表。 | 算法 | 策略项 $f(r_t)$ | 优势项 $g(A_t)$ | 正则项 $h(\text{KL}_t)$ | 训练模型数 | |------|----------------|----------------|----------------------|----------| | **DPO** | $\log r_w - \log r_l$ | 无显式优势项 | 隐含在 $\beta$ 中 | 1 (前向参与 2) | | **PPO** | $\min(r, \text{clip}(r))$ | $R - V(s)$ | $\beta \cdot \mathbb{E}[\text{KL}]$ | 2 | | **GRPO** | $\min(r, \text{clip}(r))$ | $\frac{R - \mu}{\sigma}$ | $\beta \cdot \text{KL}_t$ | 1 | | **CISPO** | $\mathrm{clip}(r, 0, \varepsilon_{max}) \cdot A_t \cdot \log \pi_\theta$ | $\frac{R - \mu}{\sigma}$ | $\beta \cdot \text{KL}_t$ | 1 | **说白了，这些 RL 算法不是割裂独立的，而是在统一优化视角下，对同一目标函数进行不同设计权衡后形成的自然变体，呈现为一种优美自洽的统一。** --- ## Ⅴ 训练结果开源 📦 #### ① PyTorch模型 ([ModelScope](https://www.modelscope.cn/models/gongjy/minimind-3-pytorch) | [HuggingFace](https://huggingface.co/jingyaogong/minimind-3-pytorch)) > 注：模型权重以实际 release 为准。并非所有训练阶段或实验分支（如 DPO、PPO、GRPO、CISPO、Agent、LoRA 等）的权重都会持续维护并单独公开；部分权重仅用于实验验证或学习用途，随着数据迭代或模型调整，逐一同步更新所有版本的必要性有限，且会带来较高的维护与训练成本。 Torch文件命名对照 - Dense: - Pretrain: `pretrain_{hidden_size}.pth` - SFT: `full_sft_{hidden_size}.pth` - DPO: `dpo_{hidden_size}.pth` - PPO: `ppo_actor_{hidden_size}.pth` - GRPO: `grpo_{hidden_size}.pth` - Agent: `agent_{hidden_size}.pth` - LoRA: `lora_xxx_{hidden_size}.pth` - MoE: - 对应同名权重在末尾追加了 `_moe` 后缀，例如：`pretrain_{hidden_size}_moe.pth`、`full_sft_{hidden_size}_moe.pth` #### ② Transformers模型 ([ModelScope](https://www.modelscope.cn/collections/MiniMind-b72f4cfeb74b47) | [HuggingFace](https://huggingface.co/collections/jingyaogong/minimind-66caf8d999f5c7fa64f399e5)) > 注：如无特殊说明，`transformers` 版本通常由 `full_sft` 权重转换而来。RL 类后训练更偏向围绕特定奖励目标优化，虽然通常能提升 reward score，但会牺牲部分通用能力和知识；这类 reward hacking / capability trade-off 在所有模型上都很难避免，更多是程度上的差异。 --- # 📌 评估 ## Ⅰ RL模型对比 基于`minimind-3 (64M)`在相同随机种子等超参下的主/客观对比，供参考： [A] minimind-3 (64M, SFT) [B] minimind-3 (64M, GRPO) [C] minimind-3 (64M, Agent-CISPO) ### 测试1：主观问答对比 ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### 测试2：轻 Agent 任务对比 一个基于 `eval_toolcall` 脚本改出来的测试，用一组数学 ToolUse 任务，对比当前 `agent` 权重和 `full_sft` 权重的表现： ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### 👉 综合评价1 从这组结果看，当前 `agent` 相比 `full_sft`，在带工具调用的轻量 Agent 任务上已经明显拉开差距了。尤其是这类“先决定要不要调工具，再把可验证结果做对”的题型里，`agent` 的成功率更高，说明 RL 之后模型在 ToolUse 这条线上确实学到了更强的调用与利用能力。 但这个提升并不是没有代价。`agent` 更适合这类轻 Agent / ToolUse 场景，并不意味着它在通用问答上也同步变强。实际体验里，这类权重在事实性问题上的稳定性通常会下降一些，知识幻觉也会更明显，更容易出现“工具任务做得更好，但开放问答反而更敢编”的现象。 所以如果任务目标是 ToolUse、轻量多步调用、可验证求解，那么当前 `agent` 会比 `full_sft` 更合适；但如果更看重日常聊天、知识问答的事实性和整体稳健性，`full_sft` 仍然会是更稳一点的选择。往大一点看，这其实也是当前业界后训练模型里很常见的一类现象：后训练往往能把某一条能力线拉得很高，但几乎都会伴随比较明显的“对齐税”。也就是说，模型在特定奖励目标上变强的同时，通常会牺牲掉一部分通用性、事实性或者自然分布下的稳定性。这个问题并不新，也很难彻底避免，本质上就是优化目标变窄之后，模型会越来越擅长“迎合当前奖励定义”，却不一定等价于整体能力无损提升。 ## Ⅱ 与其他模型对比 > 注：以下对比仅为体验参考，非严格 benchmark，样本量有限且带有主观性。 [A] minimind-3 (0.06B) [B] minimind-3-moe (0.2B-A0.06B) [C] [baby-llama2-chinese (0.2B)](https://github.com/DLLXW/baby-llama2-chinese) [D] [chatlm-mini-chinese (0.2B)](https://github.com/charent/ChatLM-mini-Chinese) ### 测试3：问答 ``` /* Detailed source-code truncated for AI context efficiency. */ ```python def quick_sort(arr): if len(arr) <= 1: return arr pivot = arr[len(arr) // 2] left = [x for x in arr if x < pivot] middle = [x for x in arr if x == pivot] right = [x for x in arr if x > pivot] return quick_sort(left) + middle + quick_sort(right) # 示例 arr = [3, 6, 8, 10, 1, 2, 1] print(quick_sort(arr)) # 输出: [1, 1, 2, 3, 6, 8, 10] ```这段代码首先检查输入数组的长度是否小于等于1。如果是，则直接返回数组。接下来是数组的长度，如果是数组，则直接返回数组。然后是数组中的元素，如果数组长度小于等于1，则直接返回数组。最后是数组中的元素，如果数组包含重复的元素，则不返回任何元素。 [B]: 快速排序是一种高效的排序算法，采用分治策略来将一个序列分为较小和较大的两个子序列，然后递归地排序这两个子序列。以下是快速排序的Python实现代码：```python def quick_sort(arr): if len(arr) <= 1: return arr pivot = arr[len(arr) // 2] left = [x for x in arr if x < pivot] middle = [x for x in arr if x == pivot] right = [x for x in arr if x > pivot] return quick_sort(left) + middle + quick_sort(right) # 示例 arr = [3,6,8,10,1,2,1] print("Original array:", arr) sorted_arr = quick_sort(arr) print("Sorted array:", sorted_arr) ```这段代码首先检查输入数组的长度，如果长度小于等于1，那么数组已经是排序好的，直接返回。然后选择一个基准值（pivot），将数组分为三部分：小于基准值的元素、等于基准值的元素和大于基准值的元素。接着递归地对小于和大于基准值的元素进行快速排序，并将结果合并。 [C]: 好的，以下是一段快速排序的Python代码```python sorted_print(sorted(sorted(sorted(sorted_print))```输出结果为：```[1, 2, 3, 4, 5, 6, 7, 8, 9]``` [D]: 快速排序是一种基于分治思想的高效排序算法，它的时间复杂度为 O(n log n)。它的基本思想是通过一趟排序将待排序序列划分为两个子序列，其中一个子序列的所有元素都比另一个子序列的所有元素小，然后对这两个子序列分别递归地进行排序。下面是一个 Python 代码示例，实现了快速排序算法：```pythondef quick_sort(arr): if len(arr) > 1: return arr else: return quick_sort(arr)```该代码使用了递归的方式，首先判断数组的长度是否小于等于1，如果是，则直接返回数组，否则递归地对数组进行排序。在递归过程中，我们首先判断数组的第一个元素是否大于等于1，如果是，则将数组中的元素与第一个元素进行比较，如果不等于1，则将数组中的元素与第二个元素进行比较，以此类推，直到所有元素都被排序。该代码的时间复杂度为 O(n log n)，是一个非常实用的排序算法。 [Q]: 截至2025年2月9日14时31分，电影《哪吒之魔童闹海》累计票房（含预售）突破78.09亿元，超过《星球大战：天行者崛起》的票房成绩，成为首部进入全球票房榜前40的亚洲电影。这一成就不仅标志着中国电影在国际市场的突破，也展示了中国动画电影的巨大潜力和市场吸引力。从这段文字总结20字的摘要给我。 [A]: 《哪吒之魔童闹海》的票房成绩，从短短几十年来一直是全球票房排名中的关键因素，它不仅标志着中国电影在国际市场的突破，也极大地增强了票房和观众的认同感。以下是对这一成就的简要总结：1. **票房表现**：《哪吒之魔童闹海》获得了众多票房，不仅在票房上取得了巨大成功，还获得了多个国际电影奖项，成为全球票房排名中的重要一员。2. **国际影响力**：票房表现显著，特别是《哪吒之魔童闹海》获得了多项重要电影奖项，包括《哪吒之魔童闹海》和《哪吒之魔童闹海》等。这些电影不仅在国际上获得了巨大成功，也吸引了全球观众的关注。3. **电影艺术价值**：电影不仅展示了中国电影的艺术魅力，还赋予了电影艺术家强大的文化价值和情感深度。《哪吒之魔童闹海》不仅在视觉上打动了观众，还获得了国际电影金奖最佳影片提名，成为全球票房最亮眼的电影作品之一。4. **市场地位**：《哪吒之魔童闘海》获得了众多国际电影奖项，不仅在国内获得了巨大成功，还吸引了来自世界各地的观众。其在国际市场上的影响力也十分强烈。5. **国际影响力**：尽管票房表现优异，但《哪吒之魔童闹海》在国际市场的表现仍然较为突出，成为全球票房排名中的重要一员。6. **国际市场**：电影在国际市场上的知名度和影响力持续增长，尤其是在国际电影节上，许多国际电影节还在进行。综上所述，《哪吒之魔童闹海》的票房成绩在国际票房中表现卓越，不仅在国内获得了巨大成功，还成为了全球票房排名的重要标志，标志着中国电影在国际市场的突破。 [B]: 20字的摘要：20字的巨大成就是中国电影在国际市场取得的首次突破，标志着中国电影在国际市场的突破。 [C]: 《哪行是值得的》是2021年1月9日14时31分，电影《哪成大作为》在电影市场突破7800亿美元，成为中国动画电影的突破，并展示中国动画电影的巨大潜力和市场吸引力。 [D]: 我是一个参数很少的AI模型🥺，知识库较少，无法直接回答您的问题，换个问题试试吧👋 ``` 🙋‍直接把以上所有问题和模型的回答丢给 GPT-5.4 Thinking，让它帮忙点评和排名打分： 具体点评 ```text ### 评分标准： - **准确性（30分）**：回答中的事实是否正确，是否存在明显的知识性错误或幻觉。 - **完整性（30分）**：回答是否涵盖了问题的核心要点，展开是否充分。 - **逻辑性（20分）**：回答是否条理清晰、前后一致，有没有自相矛盾或语义混乱。 - **代码质量（20分）**：代码是否能正确运行，实现逻辑是否清晰（仅对代码题计分）。 ### 逐模型点评： 1. **A模型（minimind-3, 0.06B）**： - **优点**：生成量充足，展开能力在这个参数量下已经不错。代码题给出了结构完整且可运行的快速排序实现，是本轮代码最好的回答之一。珠峰题也基本答对了核心信息。 - **缺点**：事实性错误比较密集——万有引力归功于爱因斯坦、长江被描述为"中国的官方名称"、海水咸度的解释完全偏离科学事实（涉及"光散射""阳光反射"等）。摘要题没有遵循20字限制，输出了大段展开。大熊猫回答虽然答对了竹子，但6个要点全是"竹子"的重复变体，信息密度极低。 - **总评**：有一定的生成和代码能力，但知识准确性是硬伤，幻觉问题突出，回答中经常出现"看起来像那么回事但细看全是编的"的现象。 2. **B模型（minimind-3-moe, 0.2B-A0.06B）**： - **优点**：回答结构相对清晰，语句通顺度在四个模型中最好。代码题实现正确且附带了示例输出，解释也比较到位。珠峰题回答准确。摘要题虽然超字数，但至少抓住了"中国电影""国际市场突破"两个关键词。 - **缺点**：知识性错误同样非常明显——长江被直接描述为"珠穆朗玛峰"，万有引力归于爱因斯坦，大熊猫的食物里出现了"海产品、鱼类、鸟类"等严重事实错误。海水咸度的解释围绕"渗透压"反复绕圈，没有触及核心原因。 - **总评**：MoE架构带来了更好的表达流畅度和结构感，但准确性问题与A模型相当。综合来看，它在"写得像不像话"这个维度上领先，但在"写得对不对"上并没有本质优势。 3. **D模型（chatlm-mini-chinese, 0.2B）**： - **优点**：知识问答表现最扎实——长江的描述基本正确（发源地、流经省区、注入东海），万有引力准确归于牛顿并引用了1687年《自然哲学的数学原理》，大熊猫主食竹子也回答正确，海水咸度的解释一开始也是对的（氯化钠、溶解盐类）。整体可读性好，不会出现明显的逻辑断裂。 - **缺点**：代码题判断条件写反（`len(arr) > 1: return arr`），导致函数完全无法工作。摘要题直接放弃回答（"我是一个参数很少的AI模型"）。珠峰和海水咸度回答后半段都出现了明显的重复退化现象。 - **总评**：知识储备在四个模型中最好，事实性问答明显领先，但代码能力是短板，生成后段容易退化为重复循环。 4. **C模型（baby-llama2-chinese, 0.2B）**： - **优点**：珠峰题回答简洁准确，大熊猫主食竹子也答对了，说明在非常基础的事实问题上还有一定能力。 - **缺点**：长江题完全答非所问（"中国是世界上最长的城市"），万有引力虽然提到了牛顿但解释混乱且自我重复，海水题答非所问（在讲水的生物学作用），代码题输出了完全不可用的代码（`sorted_print(sorted(sorted(...)))`），摘要题信息严重错乱（"哪行是值得的""7800亿美元"）。 - **总评**：基础语言能力明显不足，大部分回答要么答非所问，要么信息严重失真，在本次评测中整体垫底。 ### 总结： - **B模型**：表达最流畅、代码正确、结构感最好，但知识幻觉严重（长江=珠峰、大熊猫吃海产品），"写得顺"和"写得对"之间落差很大。 - **D模型**：知识准确性最高，在事实类问答中表现最稳定，但代码能力是明显短板，生成后段容易重复退化。 - **A模型**：与B风格接近，代码可用，但整体稳定性不如B，知识错误密度也偏高。 - **C模型**：基础能力不足，多数回答不可用，仅在最简单的事实题上偶尔答对。 ``` | 排名 | 模型 | 准确性 (30分) | 完整性 (30分) | 逻辑性 (20分) | 代码质量 (20分) | 总分 (100分) | |----|----|-----------|-----------|-----------|------------|-----------| | 1 | B | 11 | 23 | 16 | 18 | 68 | | 2 | D | 25 | 19 | 15 | 3 | 62 | | 3 | A | 10 | 21 | 13 | 17 | 61 | | 4 | C | 8 | 6 | 5 | 2 | 21 | ### 👉 综合评价2 从主观观感而言，会把 `minimind-3-moe` 放第一，`chatlm-mini-chinese` 放在第二，`minimind-3` 放在第三，`baby-llama2-chinese` 第四。`B` 虽然在知识准确性上幻觉严重（大熊猫吃海产品），但胜在表达流畅、结构清晰、代码题实现正确，综合输出质量最高；`D` 的知识储备明显领先（牛顿1687年、长江发源地等全部正确），但代码题条件写反导致完全不可用，摘要题也直接放弃，拖了不少后腿；`A` 和 `B` 风格接近，代码同样可用，但稳定性和知识准确性都不如 `B`，属于"什么都能说两句但细看全在编"的典型；`C` 则在事实性、展开能力和整体可读性上都有明显差距，仅在最简单的事实题上偶尔答对。值得注意的是，`D` 和 `A` 的总分非常接近（62 vs 61），但优劣势分布几乎是互补的：`D` 赢在知识准确性（25 vs 10），`A` 赢在代码能力（17 vs 3）。这其实也反映了小参数模型的一个典型现象——在有限的参数预算下，"说得好"和"说得对"往往很难兼得。 --- ## Ⅳ RoPE长度外推 MiniMind 支持通过 YaRN 算法进行 RoPE 位置编码的长度外推，使模型能够更稳定地处理超出训练长度的文本序列。 原生 torch 模型在使用 `eval_llm.py` 进行推理时，只需添加 `--inference_rope_scaling` 参数即可启用 RoPE 外推： ```bash python eval_llm.py --weight full_sft --inference_rope_scaling ``` 对于 `Transformers` 格式的模型，则可以在 `config.json` 中添加以下配置实现长度外推： ```json "rope_scaling": { "type": "yarn", "factor": 16.0, "original_max_position_embeddings": 2048, "beta_fast": 32.0, "beta_slow": 1.0, "attention_factor": 1.0 } ``` 下面以 MiniMind 为例，使用不同长度的《西游记》白话文本作为输入，对比启用 RoPE scaling 前后的困惑度（PPL）变化。可以看到，在长文本场景下，启用 YaRN 外推后模型的 PPL 明显下降： > MiniMind 在不同文本长度下启用 YaRN 前后的 PPL 对比 --- ## Ⅴ 客观评测 下面就到喜闻乐见的 `benchmark` 环节，这里选取了一些微型模型进行横评比较，测试集选择 C-Eval、CMMLU、ARC-Easy、PIQA、OpenBookQA、HellaSwag、Social-IQa（除了前 2 个都是英文数据集）。 测评框架选择[lm-evaluation](https://github.com/EleutherAI/lm-evaluation-harness) ```bash # 安装 git clone https://github.com/EleutherAI/lm-evaluation-harness cd lm-evaluation-harness && pip install -e . ``` ```bash # 启动测试 # 使用的数据集：ceval-valid/cmmlu/arc_easy/piqa/openbookqa/hellaswag/social_iqa # 查看支持的数据集：lm_eval ls tasks # 对于指令模型，评测时需要加上 --apply_chat_template；对于 gpt2 这类纯基座模型，则不需要。 HF_ENDPOINT=https://hf-mirror.com lm_eval --model hf --model_args pretrained="/path/to/model",dtype=auto --tasks "task" --batch_size 16 --device cpu --trust_remote_code --apply_chat_template ``` > 注：这类选择题测评通常不是让模型自由生成完整答案，而是给定题目上下文`y`和若干候选项`x`，直接比较各候选项的条件概率`p(x | y)`，并取概率最大的选项作为预测结果。若候选项只对应单个token，那么直接比较该token的预测概率即可；若候选项会分成多个token，那么更常见的做法是比较整段候选项的条件对数概率之和。这里的候选项并不一定是`A`、`B`、`C`、`D`，有些数据集也可能只有两个选项。因此从结果上看，随机作答的准确率往往就是很强的下界，而这个量级的模型也确实长期徘徊在这个附近。 MiniMind 的数据规模远小于表中其他模型，且训练比例偏向中文，因此英文表现不佳，此外默认没有专门针对这类选择题评测格式做对齐微调，所以表现会相对弱，结果仅供娱乐： | model name | from | params | zh (ceval / cmmlu) | en (arc / piqa / obqa / hellaswag / siqa) | |---|---|---|---|---| | minimind-3 | current | 64M | 24.89 / 25.38 | 28.49 / 50.65 / 23.60 / 28.28 / 34.19 | | minimind-3-moe | current | 198M | 25.48 / 24.32 | 27.74 / 50.71 / 26.20 / 27.43 / 34.03 | | minimind-3-exam | current | 64M | 30.98 / 26.12 | 35.61 / 56.26 / 24.20 / 28.40 / 34.19 | | [Steel-LLM](https://huggingface.co/gqszhanshijin/Steel-LLM) | ZhanShiJin | 1121M | 24.89 / 25.32 | 39.69 / 65.13 / 26.00 / 35.73 / 39.15 | | [gpt2-medium](https://huggingface.co/openai-community/gpt2-medium) | OpenAI | 360M | 23.18 / 25.00 | 43.60 / 66.38 / 30.20 / 39.38 / 39.10 | | [TinyLlama-1.1B](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0) | TinyLlama | 1100M | 25.71 / 25.03 | 54.80 / 74.43 / 35.60 / 60.38 / 43.09 | | [SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct) | HuggingFace | 135M | 24.44 / 24.71 | 58.50 / 68.17 / 32.80 / 43.15 / 39.46 | | [Aquila-135M](https://huggingface.co/BAAI/Aquila-135M-Instruct) | BAAI | 135M | 25.19 / 25.10 | 54.59 / 67.52 / 34.40 / 41.67 / 39.66 | **补充说明（来源/无污染/复现）** minimind-3-exam 不是更大的基座模型，也几乎没有额外注入新知识。它仅基于 minimind-3 在 [lora_exam.jsonl](https://huggingface.co/datasets/jingyaogong/minimind_dataset/blob/main/lora_exam.jsonl) 上做了一次轻量 LoRA 对齐，再将 [lora_exam_768.pth](https://huggingface.co/jingyaogong/minimind-3-pytorch/resolve/main/lora_exam_768.pth) 合并回基模后的结果。这部分数据由 ceval 与 mmlu (英文) 的 test 子集抽样构成，并做了前缀、后缀等格式增强，主要作用是对齐选择题评测中常见的上下文与候选项组织形式，而不是学习题目答案。 本节评估所用的 7 个数据集与上述对齐数据无样本交集，因此可以认为不存在数据污染。相反，若直接在有交集的数据上微调，这类小模型的分数往往会明显失真；例如 minimind-3 在有污染的 ceval / cmmlu 子集上实验跑到过约 97% 准确率，但这种结果没有参考意义。 这个实验想说明的只是：对这类评测，小模型的瓶颈未必完全在知识本身，也可能在于输入格式没有对齐。仅做少量格式对齐后，minimind-3-exam 在上面 7 个任务上平均提升约 2.9 个百分点。 # 📌 其他 ## 🔧 模型转换 * [./scripts/convert_model.py](./scripts/convert_model.py) 可用于 `torch / transformers` 两种模型格式之间的相互转换。 * 如无特殊说明，`MiniMind` 主线发布的开源模型通常以 `Transformers` 格式提供；若使用原生 `torch` 权重，请先执行 `torch2transformers` 转换。 ## 🖥️ 基于 MiniMind 的 API 服务接口 * [./scripts/serve_openai_api.py](./scripts/serve_openai_api.py) 提供了一个兼容 OpenAI API 的轻量聊天服务，便于将自己的模型接入 FastGPT、OpenWebUI、Dify 等第三方 UI。 * 当前接口额外支持 `reasoning_content`、`tool_calls`、`open_thinking` 等字段，适合直接用于 Tool Calling / Thinking 场景。 * 从 [HuggingFace](https://huggingface.co/collections/jingyaogong/minimind-66caf8d999f5c7fa64f399e5) 下载模型权重后，目录结构示例如下： ``` minimind (root dir) ├─（例如minimind-3） | ├── config.json | ├── generation_config.json | ├── model_minimind.py (可选，取决于模型导出形式) | ├── pytorch_model.bin or model.safetensors | ├── special_tokens_map.json | ├── tokenizer_config.json | ├── tokenizer.json ``` * 启动服务端 ```bash cd scripts && python serve_openai_api.py ``` * 测试服务接口 ```bash cd scripts && python chat_api.py ``` * API 请求示例（兼容 OpenAI API 格式） ```bash curl http://localhost:8998/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "model-identifier", "messages": [ { "role": "user", "content": "世界上最高的山是什么？" } ], "temperature": 0.7, "max_tokens": 1024, "stream": true, "open_thinking": true }' ``` ## [SGLang](https://github.com/sgl-project/sglang) SGLang 是高性能大模型推理引擎，支持 RadixAttention、连续批处理等优化技术，能够提供较低延迟与较高吞吐。 > ⚠️ 需要 CUDA 环境，按需使用。也可在 RL 训练脚本中选择 SGLang 作为 rollout / 推理引擎以提升训练吞吐。 以 OpenAI-compatible API server 形式启动模型： ```bash python -m sglang.launch_server --model-path /path/to/model --attention-backend triton --host 0.0.0.0 --port 8998 ``` ## [vllm](https://github.com/vllm-project/vllm) vLLM 是目前非常常用的高效推理框架，适合快速部署大模型，并在显存利用率与吞吐量之间取得较好平衡。 > ⚠️ 需要 CUDA 环境，按需使用。 以 OpenAI-compatible API server 形式启动模型： ```bash vllm serve /path/to/model --model-impl transformers --served-model-name "minimind" --port 8998 ``` ## [llama.cpp](https://github.com/ggerganov/llama.cpp) llama.cpp 是一个轻量且实用的 C++ 推理框架，可直接在命令行中使用，支持多线程推理，也支持部分 GPU 加速方案。 **目录结构**：建议将 `llama.cpp` 与模型目录放在同级路径下 ``` parent/ ├── project/ # 你的项目目录 │ ├── minimind 模型路径/ # HuggingFace 格式模型目录 │ │ ├── config.json │ │ ├── model.safetensors │ │ └── ... │ └── ... └── llama.cpp/ # llama.cpp 项目目录 ├── build/ ├── convert_hf_to_gguf.py └── ... ``` 0、参考 `llama.cpp` 官方文档完成安装（如 `cmake` 等依赖） 1、在 `convert_hf_to_gguf.py` 的 `get_vocab_base_pre` 函数末尾插入： ```python # 添加 MiniMind tokenizer 支持（此处可临时复用一个兼容项，如 qwen2） if res is None: res = "qwen2" ``` 2、将 HuggingFace 格式的 minimind 模型转换为 GGUF： ```bash # 在 llama.cpp 目录下执行，将在模型目录下生成对应的 gguf 文件 python convert_hf_to_gguf.py /path/to/minimind-model ``` 3、量化模型（可选） ```bash ./build/bin/llama-quantize /path/to/model/xxxx.gguf /path/to/model/xxxx.q8.gguf Q8_0 ``` 4、命令行推理测试 ```bash ./build/bin/llama-cli -m /path/to/model/xxxx.gguf ``` ## [ollama](https://ollama.ai) Ollama 是本地运行大模型的常用工具，支持多种开源 LLM，使用方式简洁，部署门槛较低。 1、通过 Ollama 加载自定义 GGUF 模型 在模型目录下新建 `minimind.modelfile` 文件，并写入如下配置模板： minimind.modelfile (template) ```text FROM /path/to/model/xxxx.gguf SYSTEM "你的名字叫MiniMind，你是一个乐于助人、知识渊博的AI助手。请用完整且友好的方式回答用户问题，当被问到名字时请回答MiniMind。" TEMPLATE """{{- if .Tools }}<|im_start|>system {{ if .System }}{{ .System }} {{ end }}# Tools You may call one or more functions to assist with the user query. You are provided with function signatures within XML tags: {{- range .Tools }} {"type": "function", "function": {{ .Function }}} {{- end }} For each function call, return a json object with function name and arguments within XML tags: {"name": , "arguments": } <|im_end|> {{ else if .System }}<|im_start|>system {{ .System }}<|im_end|> {{ end }} {{- range $i, $_ := .Messages }} {{- $last := eq (len (slice $.Messages $i)) 1 -}} {{- if eq .Role "user" }}<|im_start|>user {{ .Content }}<|im_end|> {{ else if eq .Role "assistant" }}<|im_start|>assistant {{ .Thinking }} {{ .Content }} {{- if .ToolCalls }} {{- range .ToolCalls }} {"name": "{{ .Function.Name }}", "arguments": {{ .Function.Arguments }}} {{- end }} {{- end }} {{- if not $last }}<|im_end|> {{ end }} {{- else if eq .Role "tool" }}<|im_start|>user {{ .Content }} <|im_end|> {{ end }} {{- if and (ne .Role "assistant") $last }}<|im_start|>assistant {{ if and $.IsThinkSet $.Think -}} {{ else -}} {{ end -}} {{ end }} {{- end }}""" PARAMETER repeat_penalty 1 PARAMETER stop "<|im_start|>" PARAMETER stop "<|im_end|>" PARAMETER temperature 0.9 PARAMETER top_p 0.9 PARAMETER num_ctx 8192 ``` 2、加载并命名本地模型 ```bash ollama create -f minimind.modelfile minimind-local ``` 3、启动推理 ```bash ollama run minimind-local ``` 📤 推送你的模型到 Ollama Hub ```bash # 1. 为本地模型重命名为你的 `ollama-account/minimind` 的 tag ollama cp minimind-local:latest your_username/minimind:latest # 2. 推送模型 ollama push your_username/minimind:latest ``` ⭐️ 也可以直接使用我提供的 Ollama 模型快速启动： ```bash ollama run jingyaogong/minimind-3 >>> 你叫什么名字 我是一个语言模型... ``` ## [MNN](https://github.com/alibaba/MNN) MNN 是面向端侧的 AI 推理引擎，支持多种开源 LLM 的轻量化部署与高性能推理。 1. 模型转换 ```bash cd MNN/transformers/llm/export # 导出 4-bit HQQ 量化的 MNN 模型 python llmexport.py --path /path/to/模型路径/ --export mnn --hqq --dst_path 模型路径-mnn ``` 2. 在 Mac 或手机端测试 ```bash ./llm_demo /path/to/模型路径-mnn/config.json prompt.txt ``` 或者下载 APP 进行测试 > 以上第三方框架的更多用法请参考对应官方文档😊 ## 👨‍💻 更多内容 * [🔗从MiniMind-LLM微调扩散语言模型](https://github.com/jingyaogong/minimind/discussions/618) * [🔗模型的generate方法说明](https://github.com/jingyaogong/minimind/discussions/611) * [🔗从 MiniMind 训练线性注意力模型](https://github.com/jingyaogong/minimind/discussions/704) # 📌 致谢 > [!NOTE] > 如果 `MiniMind` 系列项目对您有所帮助，欢迎在 GitHub 上点亮一个 ⭐ > 文档篇幅较长，难免存在疏漏之处，欢迎通过 Issues 交流反馈，或提交 PR 一起改进项目 > 您的支持与建议，都是这个项目持续迭代的重要动力！ ## 🤝[贡献者](https://github.com/jingyaogong/minimind/graphs/contributors) ## 😊鸣谢 感谢以下贡献者在训练记录、数据处理、教程整理与项目拆解等方面提供的帮助与分享： * [@ipfgao](https://github.com/ipfgao)：[🔗训练步骤记录](https://github.com/jingyaogong/minimind/issues/26) * [@WangRongsheng](https://github.com/WangRongsheng)：[🔗大型数据集预处理](https://github.com/jingyaogong/minimind/issues/39) * [@pengqianhan](https://github.com/pengqianhan)：[🔗一个简明教程](https://github.com/jingyaogong/minimind/issues/73) * [@RyanSunn](https://github.com/RyanSunn)：[🔗推理过程学习记录](https://github.com/jingyaogong/minimind/issues/75) * [@Nijikadesu](https://github.com/Nijikadesu)：[🔗以交互笔记本方式分解项目代码](https://github.com/jingyaogong/minimind/issues/213) 致谢以下优秀的论文与项目： - [https://github.com/meta-llama/llama3](https://github.com/meta-llama/llama3) - [https://github.com/karpathy/llama2.c](https://github.com/karpathy/llama2.c) - [https://github.com/DLLXW/baby-llama2-chinese](https://github.com/DLLXW/baby-llama2-chinese) - [DeepSeek-V2](https://arxiv.org/abs/2405.04434) - [https://github.com/charent/ChatLM-mini-Chinese](https://github.com/charent/ChatLM-mini-Chinese) - [https://github.com/wdndev/tiny-llm-zh](https://github.com/wdndev/tiny-llm-zh) - [Mistral-MoE](https://arxiv.org/pdf/2401.04088) - [https://github.com/Tongjilibo/build_MiniLLM_from_scratch](https://github.com/Tongjilibo/build_MiniLLM_from_scratch) - [https://github.com/jzhang38/TinyLlama](https://github.com/jzhang38/TinyLlama) - [https://github.com/AI-Study-Han/Zero-Chatgpt](https://github.com/AI-Study-Han/Zero-Chatgpt) - [https://github.com/xusenlinzy/api-for-open-llm](https://github.com/xusenlinzy/api-for-open-llm) - [https://github.com/HqWu-HITCS/Awesome-Chinese-LLM](https://github.com/HqWu-HITCS/Awesome-Chinese-LLM) ## 🫶支持者 ## 🎉 MiniMind 相关成果 本模型抛砖引玉地促成了一些可喜成果的落地，感谢研究者们的认可： - ECG-Expert-QA: A Benchmark for Evaluating Medical Large Language Models in Heart Disease Diagnosis [[arxiv](https://arxiv.org/pdf/2502.17475)] - Binary-Integer-Programming Based Algorithm for Expert Load Balancing in Mixture-of-Experts Models [[arxiv](https://arxiv.org/pdf/2502.15451)] - LegalEval-Q: A New Benchmark for The Quality Evaluation of LLM-Generated Legal Text [[arxiv](https://arxiv.org/pdf/2505.24826)] - On the Generalization Ability of Next-Token-Prediction Pretraining [[ICML 2025](https://openreview.net/forum?id=hLGJ1qZPdu)] - 《从零开始写大模型：从神经网络到Transformer》王双、牟晨、王昊怡 编著 - 清华大学出版社 - FedBRB: A Solution to the Small-to-Large Scenario in Device-Heterogeneity Federated Learning [[TMC 2025](https://ieeexplore.ieee.org/abstract/document/11168259)] - SKETCH: Semantic Key-Point Conditioning for Long-Horizon Vessel Trajectory Prediction [[arxiv](https://arxiv.org/pdf/2601.18537)] - A Built-in Crypto Expert for Artificial Intelligence: How Far is the Horizon? [[IACR ePrint 2026](https://eprint.iacr.org/2026/411.pdf)] - RetryTrigger: Intelligent Inference Duplication for Enhancing LLM Resilience to Hardware Transient Faults [[FITEE 2026](https://ieeexplore.ieee.org/abstract/document/11479682)] - 进行中... # 🎓 引用 如果 `MiniMind` 对您的研究或工作有所帮助，欢迎引用： ```bibtex @misc{minimind, title = {MiniMind: Train a Tiny LLM from Scratch}, author = {Jingyao Gong}, year = {2024}, url = {https://github.com/jingyaogong/minimind}, note = {GitHub repository, accessed 2026} } ``` # ⚖️ 开源协议 本项目采用 [Apache License 2.0](LICENSE) 开源协议。 --- ## File: README_en.md * This open-source project aims to train MiniMind, an ultra-small language model with about 64M parameters, entirely from scratch with only about RMB 3 in cost and 2 hours of training time. * The MiniMind series is intentionally lightweight. The smallest model on the main branch is about $\frac{1}{2700}$ the size of GPT-3, making full training and reproduction feasible even on ordinary personal GPUs. * The project provides a minimalist model architecture and an end-to-end LLM training pipeline, covering MoE, data cleaning, pretraining, Supervised Fine-Tuning (SFT), LoRA, RLHF (DPO), RLAIF (PPO / GRPO / CISPO), Tool Use, Agentic RL, Adaptive Thinking, and Model Distillation. * MiniMind has also been extended to a vision model [MiniMind-V](https://github.com/jingyaogong/minimind-v), a multimodal Omni model [MiniMind-O](https://github.com/jingyaogong/minimind-o), a diffusion language model (MiniMind-dLM), and a linear attention model (MiniMind-Linear). See [Discussion](https://github.com/jingyaogong/minimind/discussions) for details. * All core algorithms are implemented directly in native PyTorch, without relying on high-level abstractions from third-party libraries. * MiniMind is both an end-to-end open-source reproduction of the LLM training pipeline and a hands-on tutorial for learning how LLMs are built. * We hope this project can provide a reproducible, understandable, and extensible starting point for more people, share the joy of creation, and help move the broader AI community forward. > Note: This project is released under the Apache 2.0 license and is completely free. "2 hours" refers to the measured time for running `1 epoch` of the SFT stage on a single NVIDIA 3090, while "RMB 3" refers to the corresponding GPU rental cost. --- --- # 📌 Project Introduction The emergence of Large Language Models (LLMs) has drawn unprecedented global attention to AI. ChatGPT, DeepSeek, Qwen, and many other models have impressed people with their remarkable performance, making the impact of this technological wave feel very real. However, models with tens or hundreds of billions of parameters are not only difficult to train on personal devices, but often out of reach even for deployment. Opening the "black box" of large models and truly understanding how they work internally should have been an exciting thing. Unfortunately, most explorations eventually stop at applying techniques such as LoRA to fine-tune existing large models on a few new instructions or specific tasks. This is more like teaching Newton how to use a 21st-century smartphone — interesting, but not quite the original goal of understanding the essence of physics. At the same time, third-party LLM frameworks and toolkits such as `transformers` / `trl` / `peft` often expose only highly abstract interfaces. With just a dozen lines of code, one can complete the entire pipeline of "load model + load dataset + inference + reinforcement learning" training. This kind of efficient encapsulation is convenient, but it also separates developers from the underlying implementation to some extent, reducing the opportunity to deeply understand the core code of LLMs. I believe that "building an airplane from Lego bricks yourself is far more exciting than flying in first class". A more practical problem is that the internet is also filled with paid courses and marketing content, where so-called AI tutorials are wrapped in flawed and half-understood explanations. For this reason, the original intention of this project is to lower the learning barrier of LLMs as much as possible, so that everyone can start from understanding every line of code and train a tiny language model by hand from scratch. Yes, **training from scratch**, not merely staying at the **inference** level. With a server cost of less than RMB 3, you can personally experience the full process of building a language model from 0 to 1. 😊 Let's share the joy of creation together! --- #### 🎉 This Project Includes the Following - Provides the full MiniMind-LLM architecture implementation (Dense + MoE), aligned with the `Qwen3 / Qwen3-MoE` ecosystem. - Provides the tokenizer and tokenizer training code, supporting template tokens such as ``, ``, ``, etc. - Covers end-to-end training pipelines including pretraining, SFT, LoRA, RLHF-DPO, RLAIF (PPO / GRPO / CISPO), Tool Use, Agentic RL, Adaptive Thinking, and Model Distillation. - Provides open-source data for all stages, covering collected, distilled, cleaned, and deduplicated high-quality datasets. - Key training algorithms and core modules are all implemented from scratch, without relying on third-party framework wrappers. - Compatible with mainstream frameworks such as `transformers`, `trl`, `peft`, as well as commonly used inference engines like `llama.cpp`, `vllm`, `ollama`, and training frameworks like `Llama-Factory`. - Supports single-node single-GPU and single-node multi-GPU training (DDP, DeepSpeed), wandb / swanlab visualization, and dynamic training pause/resume. - Supports evaluation on third-party benchmark suites such as C-Eval, C-MMLU, OpenBookQA, etc., and supports RoPE long context extrapolation through YaRN. - Provides a lightweight OpenAI-compatible API server for integration with third-party Chat UIs such as FastGPT and Open-WebUI, with support for `reasoning_content`, `tool_calls`, and `open_thinking`. - Provides a minimalist chat WebUI based on Streamlit, supporting thinking display, tool selection, and multi-turn Tool Call. - Includes experimental extensions: diffusion language model ([dLM](https://github.com/jingyaogong/minimind/discussions/618)) and linear attention model ([Linear Attention](https://github.com/jingyaogong/minimind/discussions/704)), both of which can be further trained from the main autoregressive model. #### 🎉 Released Model List | Model | Parameters | Release | |------|--------|---------| | minimind-3 | 64M | 2026.04.01 | | minimind-3-moe | 198M-A64M | 2026.04.01 | | minimind2-small | 26M | 2025.04.26 | | minimind2-moe | 145M | 2025.04.26 | | minimind2 | 104M | 2025.04.26 | | minimind-v1-small | 26M | 2024.08.28 | | minimind-v1-moe | 4×26M | 2024.09.17 | | minimind-v1 | 108M | 2024.09.01 | --- #### 📝 Changelog **🔥 2026-04-01** - Released `minimind-3` / `minimind-3-moe`: comprehensive updates to structure, Tokenizer, training pipeline, inference interface, and default configuration - Main branch structure aligned with `Qwen3 / Qwen3-MoE` ecosystem: Dense approximately `64M`, MoE approximately `198M-A64M`, and removed shared expert design - Default training data switched to `pretrain_t2t(_mini).jsonl`, `sft_t2t(_mini).jsonl`, `rlaif.jsonl`, `agent_rl.jsonl`, and `agent_rl_math.jsonl` - Removed standalone `train_reason.py`; thinking capability is now unified through `chat_template + ` and `open_thinking` adaptive switch control - `toolcall` capability has been merged into `sft_t2t / sft_t2t_mini` main branch data, default `full_sft` already has basic Tool Call capability; also added inference examples such as `scripts/chat_api.py` - Added native `Agentic RL` training script `train_agent.py`, supporting `GRPO / CISPO` in multi-turn Tool-Use scenarios - RLAIF / Agentic RL training pipeline completed `rollout engine` decoupling, supporting more flexible switching of generation backends - `serve_openai_api.py` and `web_demo.py` added `reasoning_content` / `tool_calls` / `open_thinking` support - Tokenizer updated based on `BPE + ByteLevel`, with new tool call and thinking tokens, reserved buffer tokens for future extension - Added LoRA weight merging and export pipeline, can merge base model and LoRA weights into new complete model weights via `scripts/convert_model.py` - Structure diagram resources updated, README extensively updated **2025-10-24** - 🔥 Added RLAIF training algorithms: PPO, GRPO, SPO (natively implemented from scratch) - Added checkpoint resume functionality: supports automatic training recovery, cross-GPU-count recovery, wandb record continuity - Added RLAIF dataset: rlaif-mini.jsonl (randomly sampled 10,000 entries from SFT data); simplified DPO dataset, added Chinese data - Added YaRN algorithm: supports RoPE long context extrapolation, improving long sequence processing capability - Adaptive Thinking: Reason model optionally enables chain of thought - chat_template fully supports Tool Calling and Reasoning tags (``, ``, etc.) - Added complete RLAIF chapter, training curve comparison, folded algorithm principle explanations - [SwanLab](https://swanlab.cn/) replaces WandB (domestic access friendly, API fully compatible) - Standardized all code & fixed some known bugs **2025-04-26** - Major update - For compatibility needs, visit [🔗Old Repository Content🔗](https://github.com/jingyaogong/minimind/tree/7da201a944a90ed49daef8a0265c959288dff83a). - MiniMind model parameters completely renamed, aligned with Transformers library models (unified naming). - generate method refactored, inheriting from GenerationMixin class. - 🔥Supports popular third-party ecosystems such as llama.cpp, vllm, ollama. - Standardized code and directory structure. - Changed vocabulary `` -> `<|im_start|><|im_end|>` ```text To be compatible with third-party inference frameworks llama.cpp, vllm, this update comes with some considerable costs. This update no longer supports "directly" loading old models from before 25-04-26 for inference. Due to differences between Llama's positional encoding method and minimind's, QK values differ after mapping to the Llama model. The minimind2 series old models were all recovered through weight mapping + (fine-tuning) QKVO linear layer calibration. After this update, maintenance for the entire `minimind-v1` series will be discontinued and taken offline from the repository. ``` **More...** **2025-02-09** - Major update since release, Release minimind2 Series. - Code almost entirely refactored, using a more concise and clear unified structure. For compatibility needs with old code, visit [🔗Old Repository Content🔗](https://github.com/jingyaogong/minimind/tree/6e9cd28ef9b34a0a10afbdf6f59e65cb6e628efb). - Eliminated data preprocessing steps. Unified dataset format, switched to `jsonl` format to avoid dataset download confusion issues. - minimind2 series significantly improved performance compared to MiniMind-V1. - Minor issues: {kv-cache implementation more standard, MoE load balancing loss now considered, etc.} - Provides training solution for migrating models to private datasets (medical model, self-awareness examples). - Streamlined pretraining dataset and significantly improved pretraining data quality, greatly reduced time needed for individual quick training, reproducible in 2 hours on a single 3090! - Updated: LoRA fine-tuning decoupled from peft wrapper, LoRA process implemented from scratch; DPO algorithm natively implemented from scratch using PyTorch; model white-box distillation natively implemented. - minimind2-DeepSeek-R1 series distilled models born! - minimind2 has certain English language capability! - Updated benchmark test performance results of minimind2 vs third-party models on more LLM leaderboards. **2024-10-05** - Extended multimodal capability for MiniMind --- Vision - Visit the sibling project [minimind-v](https://github.com/jingyaogong/minimind-v) for details! **2024-09-27** - 09-27 updated pretrain dataset preprocessing method, to ensure text integrity, abandoned preprocessing into .bin format for training (slight sacrifice in training speed). - Currently the pretrain preprocessed file is named: pretrain_data.csv. - Removed some redundant code. **2024-09-17** - Updated minimind-v1-moe model - To prevent ambiguity, mistral_tokenizer is no longer used for tokenization, all using custom minimind_tokenizer as the tokenizer. **2024-09-01** - Updated minimind-v1 (108M) model, using minimind_tokenizer, pretrain epochs 3 + SFT epochs 10, more thorough training, stronger performance. - Project deployed to ModelScope Creative Space, can be experienced at this website: - [🔗ModelScope Online Demo🔗](https://www.modelscope.cn/studios/gongjy/minimind) **2024-08-27** - Project first open-sourced --- # 📌 Quick Start My hardware and software configuration (for reference) * CPU: Intel(R) Core(TM) i9-10980XE CPU @ 3.00GHz * RAM: 128 GB * GPU: NVIDIA GeForce RTX 3090 (24GB) * 8 * Ubuntu==20.04 * CUDA==12.2 * Python==3.10.16 * [requirements.txt](./requirements.txt) ## Step 0 ```bash # Clone repository and install dependencies git clone --depth 1 https://github.com/jingyaogong/minimind cd minimind && pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple ``` ## Ⅰ 🚀 Model Inference ### 1' Download Model In the project root directory: ```bash # Method 1 modelscope download --model gongjy/minimind-3 --local_dir ./minimind-3 # Method 2 git clone https://huggingface.co/jingyaogong/minimind-3 ``` ### 2' CLI Inference ```bash # Method 1: Using Transformers format model python eval_llm.py --load_from ./minimind-3 # Method 2: Based on PyTorch model (ensure corresponding weights are in the ./out directory) python eval_llm.py --load_from ./model --weight full_sft ``` ### 3' (Optional) WebUI ```bash # May need `python>=3.10`, install `pip install streamlit` # ⚠️ You must first copy the transformers-format model folder into ./scripts/ (e.g.: cp -r minimind-3 ./scripts/minimind-3). The web_demo script will auto-scan subdirectories containing weight files; it will throw an error if none are found. cd scripts && streamlit run web_demo.py ``` ### 4' (Optional) Third-party Inference Frameworks ```bash # ollama ollama run jingyaogong/minimind-3 # vllm vllm serve /path/to/model --served-model-name "minimind" ``` ## Ⅱ 🛠️ Model Training Note: Confirm Torch's available backend in advance ```python import torch print(torch.cuda.is_available()) ``` If you plan to use CUDA for training, it is recommended to first confirm whether the current environment has correctly recognized the GPU. If `cuda` is not available, you can still choose `CPU` or `MPS` to run based on your device, but training speed and compatibility will differ significantly. If you need to install or switch PyTorch versions, refer to [torch_stable](https://download.pytorch.org/whl/torch_stable.html) and [this link](https://blog.csdn.net/weixin_45456738/article/details/141029610?ops_request_misc=&request_id=&biz_id=102&utm_term=%E5%AE%89%E8%A3%85torch&utm_medium=distribute.pc_search_result.none-task-blog-2~all~sobaiduweb~default-2-141029610.nonecase&spm=1018.2226.3001.4187) ### 1' Download Data Download the required data files from the [dataset download link](https://www.modelscope.cn/datasets/gongjy/minimind_dataset/files) provided below, and place them in the `./dataset` directory > Currently, by default, you only need to download `pretrain_t2t_mini.jsonl` and `sft_t2t_mini.jsonl` to quickly reproduce the `MiniMind Zero` dialogue model. For other use cases, several data combinations are provided below and can be selected according to your task objectives and GPU resources. ### 2' Start Training 💡 Checkpoint Pause and Resume All training scripts support checkpoint saving. After adding the `--from_resume 1` parameter, the script will automatically detect and resume training progress: ```bash python train_pretrain.py --from_resume 1 python train_full_sft.py --from_resume 1 # ... ``` **Checkpoint Resume Instructions:** - The training process automatically saves complete checkpoints (model, optimizer, training progress, etc.) in the `./checkpoints/` directory - Checkpoint file naming: `__resume.pth` (e.g., `full_sft_512_resume.pth`) - Supports recovery across different GPU counts (automatically adjusts step) - Supports wandb training record continuity (automatically resumes the same run) > Suitable for long-duration training or unstable environments, no need to worry about progress loss due to training interruption #### 2.1 Pretraining (Required) ```bash cd trainer && python train_pretrain.py ``` > After training, `out/pretrain_*.pth` will be produced as output weights (where `*` is the model dimension, default `768`) #### 2.2 Instruction Fine-tuning (Required) ```bash cd trainer && python train_full_sft.py ``` > After training, `out/full_sft_*.pth` will be produced as output weights (where `full` indicates full-parameter fine-tuning) #### 2.3 Test Trained Model (Optional) Ensure the model `*.pth` files to be tested are located in the `./out/` directory; you can also go directly to [here](https://www.modelscope.cn/models/gongjy/minimind-3-pytorch/files) to download my pre-trained `*.pth` weights. ```bash python eval_llm.py --weight full_sft ``` > `--weight` is used to specify the weight name prefix, such as `pretrain`, `full_sft`, etc.; for more parameters, refer directly to `eval_llm.py` Note: Other Information 1. All training scripts are natively implemented based on PyTorch and support multi-GPU acceleration. 2. If your device has `N (N > 1)` GPUs, you can start single-machine `N`-GPU training as follows (DDP, also supports extension to multi-machine multi-GPU): ```bash torchrun --nproc_per_node N train_xxx.py ``` 3. You can enable wandb to record the training process as needed. ```bash ... train_xxx.py --use_wandb ``` After June `2025`, domestic network environments in China typically cannot directly connect to WandB. MiniMind currently defaults to using [SwanLab](https://swanlab.cn/) as the training visualization tool, whose interface is basically compatible with WandB; usually you only need to replace `import wandb` with `import swanlab as wandb`, and other usage remains largely unchanged. --- # 📌 Data Introduction ## Ⅰ Tokenizer A tokenizer can be roughly understood as a "dictionary" used by LLMs, responsible for mapping natural language to token ids and decoding token ids back to text; the project also provides `train_tokenizer.py` as a vocabulary training example. It is not recommended to retrain the tokenizer, because once the vocabulary and segmentation rules change, model weights, data formats, inference interfaces, and community ecosystem compatibility will all be affected, and the resulting model will also be harder to share. Meanwhile, the tokenizer also affects metrics like PPL that are calculated per token, so when comparing across tokenizers, BPB (Bits Per Byte) is often a more comparable metric. See [this article](https://skeptric.com/perplexity/). For small models like MiniMind, vocabulary size also directly affects the parameter proportion of the embedding and output layers, so keeping the vocabulary compact is usually a more appropriate trade-off. Tokenizer Introduction The tokenizer vocabulary sizes of powerful third-party open-source models such as Yi, Qwen2, ChatGLM, Mistral, and Llama 3 are as follows: | Tokenizer Model | Vocabulary Size | Source | | --- | --- | --- | | Yi | 64,000 | 01.AI (China) | | Qwen2 | 151,643 | Alibaba Cloud (China) | | ChatGLM | 151,329 | Zhipu AI (China) | | Mistral | 32,000 | Mistral AI (France) | | Llama 3 | 128,000 | Meta (USA) | | MiniMind | 6,400 | Custom | > The main branch uses `minimind_tokenizer` consistently to avoid historical version ambiguity and control overall size, and no longer maintains the `mistral_tokenizer` version. Although `minimind_tokenizer` has a vocabulary of only `6400`, and its encoding/decoding efficiency is weaker than more Chinese-friendly tokenizers like `qwen2` and `glm`, it significantly reduces the parameter share of the embedding and output layers, which better fits the size constraints of small models like MiniMind. In practice, this tokenizer has not noticeably caused rare-word decoding failures and remains stable enough for general use. Therefore, the main branch uses this vocabulary consistently rather than maintaining additional tokenizer forks. ## Ⅱ Pretrain Data The pretraining data for the current `MiniMind-3` main branch is `pretrain_t2t.jsonl` / `pretrain_t2t_mini.jsonl`. These two datasets have been organized into a unified `text -> next token prediction` training format, aiming to balance under limited compute: - Text quality; - Length distribution; - Chinese-English mixed capability; - Template alignment with subsequent SFT / Tool Calling / RLAIF stages. Data sources include but are not limited to general text corpora, curated dialogue corpora, distillation corpora, and various datasets available under **permissive open-source licenses**; main branch data enters training only after cleaning, deduplication, length control, and format unification. Data sources include: [Craftsman LLM Dataset](https://www.modelscope.cn/datasets/deepctrl/deepctrl-sft-data), [Magpie-Align](https://www.modelscope.cn/organization/Magpie-Align), and other public data sources. Among them: - `pretrain_t2t_mini.jsonl` is intended for quick reproduction; - `pretrain_t2t.jsonl` is intended for full training of the `MiniMind-3` main branch model. The file format is as follows: ```jsonl {"text": "如何才能摆脱拖延症？治愈拖延症并不容易，但以下建议可能有所帮助。"} {"text": "清晨的阳光透过窗帘洒进房间，桌上的书页被风轻轻翻动。"} {"text": "Transformer 通过自注意力机制建模上下文关系，是现代大语言模型的重要基础结构。"} ``` ## Ⅲ SFT Data The SFT data for the current `MiniMind-3` main branch is `sft_t2t.jsonl` / `sft_t2t_mini.jsonl`. Compared with earlier `sft_512 / sft_1024 / sft_2048` schemes, the current version places more emphasis on: - Unified templates; - Better suited for mixed training of dialogue + thinking tags + Tool Calling; - Minimizing data preprocessing forks, reducing reproduction costs. Its data sources include but are not limited to high-quality instruction-following data, public dialogue data, model-distilled synthetic data, and permissively licensed open-source datasets; before entering the `t2t` main branch, they are unified into the multi-turn dialogue format used by the current repository. The current main branch also contains a large amount of synthetic data, such as approximately `100K` `tool call` entries synthesized from `qwen3-4b`, as well as `reasoning` data from the `qwen3` series. Major community sources include: [Craftsman LLM Dataset](https://www.modelscope.cn/datasets/deepctrl/deepctrl-sft-data), [Magpie-Align](https://www.modelscope.cn/organization/Magpie-Align), [R1-Distill-SFT](https://www.modelscope.cn/datasets/AI-ModelScope/R1-Distill-SFT), [COIG](https://huggingface.co/datasets/BAAI/COIG), [Step-3.5-Flash-SFT](https://huggingface.co/datasets/stepfun-ai/Step-3.5-Flash-SFT), etc. Published versions ensure that data sources and processing pipelines comply with the transitivity constraints of corresponding open-source licenses, and adhere to Apache-2.0, CC-BY-NC-2.0, and other related license requirements. Among them: - `sft_t2t_mini.jsonl`: suitable for quickly training a dialogue model; - `sft_t2t.jsonl`: suitable for fully reproducing the main branch version; - `toolcall` capability has already been merged into the main branch SFT data. All SFT files follow the same format, including dialogue and Tool Use data: ```jsonl { "conversations": [ {"role": "user", "content": "你好"}, {"role": "assistant", "content": "你好！"}, {"role": "user", "content": "再见"}, {"role": "assistant", "content": "再见！"} ] } { "conversations": [ {"role": "system", "content": "# Tools ...", "tools": "[...]"}, {"role": "user", "content": "把'你好世界'翻译成english"}, {"role": "assistant", "content": "", "tool_calls": "[{\"name\":\"translate_text\",\"arguments\":{\"text\":\"你好世界\",\"target_language\":\"english\"}}]"}, {"role": "tool", "content": "{\"translated_text\":\"Hello World\"}"}, {"role": "assistant", "content": "Hello World"} ] } ``` ## Ⅳ RL Data The RL data for the current `MiniMind` main branch is `dpo.jsonl`, sampled from [DPO-En-Zh-20k](https://huggingface.co/datasets/llamafactory/DPO-En-Zh-20k). In the main branch, these samples are reorganized into the preference-learning format used by this repository for reward-model training or preference optimization. Here, `chosen` represents the preferred response, and `rejected` represents the weaker response. The `dpo.jsonl` data format is ```json { "chosen": [ {"content": "Q", "role": "user"}, {"content": "good answer", "role": "assistant"} ], "rejected": [ {"content": "Q", "role": "user"}, {"content": "bad answer", "role": "assistant"} ] } ``` Besides this, other RL data maintains the same format as SFT data, typically filtered from SFT data by total length and dialogue turns, with the last `assistant` position left blank for continuation during the rollout stage. ## Ⅴ MiniMind Training Dataset > [!NOTE] > The core datasets needed for the current main branch training have been open-sourced, so you do not need to preprocess large-scale datasets yourself. MiniMind training dataset download links: [ModelScope](https://www.modelscope.cn/datasets/gongjy/minimind_dataset/files) | [HuggingFace](https://huggingface.co/datasets/jingyaogong/minimind_dataset/tree/main) > No need to clone everything, you can download individual files as needed Place the downloaded dataset files in the `./dataset/` directory (✨ indicates recommended essentials) ```bash ./dataset/ ├── agent_rl.jsonl (86MB) ├── agent_rl_math.jsonl (18MB) ├── dpo.jsonl (53MB) ├── pretrain_t2t_mini.jsonl (1.2GB, ✨) ├── pretrain_t2t.jsonl (10GB) ├── rlaif.jsonl (24MB, ✨) ├── sft_t2t_mini.jsonl (1.6GB, ✨) └── sft_t2t.jsonl (14GB) ``` Note: Brief Introduction to Each Dataset * `agent_rl.jsonl` -- Agentic RL main branch training data, for `train_agent.py` multi-turn Tool-Use / CISPO / GRPO training * `agent_rl_math.jsonl` -- Agentic RL pure math supplementary data, suitable for multi-turn reasoning/tool-use scenarios with final verification targets (for RLVR) * `dpo.jsonl` -- RLHF stage preference training data (DPO) * `pretrain_t2t_mini`✨ -- `minimind-3` lightweight pretraining data, suitable for quick reproduction (recommended setting `max_seq_len≈768`) * `pretrain_t2t` -- `minimind-3` main branch pretraining data (recommended setting `max_seq_len≈380`) * `rlaif.jsonl`✨ -- RLAIF training dataset, for PPO/GRPO/CISPO and other reinforcement learning algorithm training * `sft_t2t_mini.jsonl`✨ -- `minimind-3` lightweight SFT data (for quickly training a Zero model), recommended setting `max_seq_len≈768`, with a portion of Tool Call samples already mixed in * `sft_t2t.jsonl` -- `minimind-3` main branch SFT data, suitable for full reproduction, with Tool Call samples also mixed in The training parameter `max_seq_len` currently refers to token length, not absolute character count. This project's tokenizer has approximately `1.5~1.7 characters/token` for Chinese text, and a compression ratio of `4~5 characters/token` for pure English, with fluctuations depending on data distribution. The "maximum length" annotated in dataset names is in character count; a string of 100 characters can be roughly converted to approximately `100/1.5≈67` tokens. For example: * Chinese: `白日依山尽` (5 characters) may be split into [`白日`,`依`,`山`,`尽`] 4 tokens; * English: `The sun sets in the west` (24 characters) may be split into [`The `,`sun `,`sets `,`in `,`the`,`west`] 6 tokens The "recommended settings" provide rough estimates of the maximum token length for each dataset. Note that `max_seq_len` can be adjusted aggressively or conservatively, but both directions have side effects: samples shorter than `max_seq_len` waste compute due to padding, while samples longer than `max_seq_len` lose information due to truncation. In practice, it is enough to balance compute efficiency and semantic completeness. > MiniMind main branch training data composition and recommended combination diagram Instructions & Recommended Training Schemes * `minimind-3` main branch recommends using the staged training combination of `pretrain_t2t` + `sft_t2t` + `rlaif/agent_rl`. * For the fastest speed to implement a Zero model from scratch, it is recommended to use the data combination of `pretrain_t2t_mini.jsonl` + `sft_t2t_mini.jsonl` * Those with sufficient compute resources or who care more about performance are recommended to fully reproduce `minimind-3`; those with only a single GPU or who prioritize quick reproduction are strongly recommended to use the mini combination. * The current `sft_t2t / sft_t2t_mini` already has Tool Call data mixed in, so there is usually no need for an additional separate round of Tool Calling supervised fine-tuning. # 📌 Model ## Structure `minimind-3` Dense uses a Transformer Decoder-Only architecture, with its overall configuration aligned with the `Qwen3` ecosystem for easier conversion to `transformers / llama.cpp / ollama / vllm`: * Uses Pre-Normalization (Pre-Norm) + RMSNorm. * Uses SwiGLU activation function. * Uses RoPE rotary positional encoding, with YaRN extrapolation support. * `q_heads=8`, `kv_heads=4`, `max_position_embeddings=32768`, `rope_theta=1e6`. `minimind-3-moe` extends MoE feed-forward layers on the same structure, with implementation compatible with `Qwen3-MoE` style configuration (removing shared expert). * The current default configuration is `4 experts / top-1 routing`, to achieve higher capacity with lower active parameters. * As the number of experts increases, training can become much slower than a dense model of similar size. This may seem counterintuitive given the common claim that "MoE inference is faster", but in training, tokens are first bucketed by expert and then forwarded separately. In a native PyTorch implementation, kernel launch and scheduling overhead quickly become significant. This usually requires fused MoE kernels or specialized libraries such as `Triton`, `DeepSpeed-MoE`, or `Megatron-LM` for optimization. MiniMind keeps the implementation in native PyTorch for portability, so this is a practical trade-off. Under the current implementation, the `4 experts / top-1` configuration is only about `50%` slower than the dense model. The `minimind-3` series structure is shown below: To modify model configuration, see [./model/model_minimind.py](./model/model_minimind.py). Reference model parameter versions are shown in the table below: | Model Name | params | len_vocab | max_pos | rope_theta | n_layers | d_model | kv_heads | q_heads | note | |------------|--------|-----------|---------|------------|----------|---------|----------|---------|------| | minimind-3 | 64M | 6400 | 32768 | 1e6 | 8 | 768 | 4 | 8 | Dense | | minimind-3-moe | 198M-A64M | 6400 | 32768 | 1e6 | 8 | 768 | 4 | 8 | 4 experts / top-1 | | minimind2-small | 26M | 6400 | 32768 | 1e6 | 8 | 512 | 2 | 8 | Historical version | | minimind2-moe | 145M | 6400 | 32768 | 1e6 | 8 | 640 | 2 | 8 | Historical version | | minimind2 | 104M | 6400 | 32768 | 1e6 | 16 | 768 | 2 | 8 | Historical version | ## Model Configuration Regarding LLM parameter configuration, [MobileLLM](https://arxiv.org/pdf/2402.14905) has conducted a very representative systematic study on small models. For ~100M-level models like MiniMind, the trade-off between `d_model` and `n_layers` is not just a parameter allocation issue, but also directly affects training stability and final performance. The current `minimind-3` main branch uses `dim=768, n_layers=8`, which is essentially an engineering trade-off: shallower networks train faster, while `dim` is still large enough to avoid a severe representation bottleneck, giving a reasonable balance between training efficiency, stability, and final performance. View Detailed Explanation Scaling laws often behave differently in the small-model regime. The main architectural parameters that determine the Transformer parameter scale are usually `d_model` and `n_layers`: * `d_model`↑ + `n_layers`↓ -> wide and shallow * `d_model`↓ + `n_layers`↑ -> narrow and deep Classic scaling laws emphasize the roles of training data size, parameter count, and training steps, and often downplay architectural differences. In the small-model regime, however, this conclusion does not always hold. One core observation from MobileLLM is that, under a fixed parameter budget, depth is often more important than width. Compared with wide and shallow models, narrow and deep models tend to learn abstract concepts more effectively. For example, when the parameter count is fixed at `125M` or `350M`, narrow models with `30~42` layers typically outperform wide models with around `12` layers, showing similar trends across benchmarks such as commonsense reasoning, QA, and reading comprehension. This is consistent with MiniMind's own experiments around `d_model` and `n_layers`. However, "narrow" also has a lower bound: when `d_model < 512`, the representation bottleneck becomes much more pronounced, and adding extra layers is often insufficient to compensate for an overly small `d_head` under a fixed `q_head`. Conversely, when `d_model > 1536`, adding layers is often more cost-effective than further increasing width, and tends to yield better parameter-performance returns. For reference, GPT-3's parameter settings are as follows: --- # 📌 Experiments ## Ⅰ Training Cost - **Time unit**: hours (h) - **Cost unit**: CNY (￥); `7￥ ≈ 1 USD` - **3090 rental price**: approximately `1.3￥/h` (actual prices may vary) - **Note**: The following results are empirical estimates for the `minimind` model on a single `3090` GPU, intended to make the training cost easy to estimate | Model Name | params | pretrain_t2t_mini | sft_t2t_mini | toolcall | RLAIF | |------------|--------|-------------------|--------------|----------|-------| | minimind-3 | 64M | ≈1.21h≈1.57￥ | ≈1.10h≈1.43￥ | ≈0.9h≈1.17￥ | ≈1.1h≈1.43￥ | | minimind-3-moe | 198M-A64M | ≈1.69h≈2.20￥ | ≈1.54h≈2.00￥ | ≈1.26h≈1.64￥ | ≈1.54h≈2.00￥ | --- Training Cost Summary & Prediction > `minimind-3` >> `pretrain_t2t_mini` + `sft_t2t_mini` Single `3090` GPU, `1 epoch` estimated approximately `2.31` hours, cost approximately `3.0` CNY Can train a `minimind-3 Zero` dialogue model from scratch. > `minimind-3-moe` >> `pretrain_t2t_mini` + `sft_t2t_mini` Single `3090` GPU, `1 epoch` estimated approximately `3.23` hours, cost approximately `4.2` CNY Can quickly obtain the basic dialogue version of `minimind-3-moe`. > The above values are estimates and are meant only as a rough reference for training cost. On a single `NVIDIA 3090`, `minimind zero` can still be trained from scratch in roughly `2` hours, making the training process accessible to individual developers. With a higher-end multi-GPU setup such as `8x H100`, the total training time can be reduced to minutes. MiniMind has always aimed to keep LLM training reproducible, hands-on, and easy to iterate on. Low-cost quick reproduction is not a gimmick; below is an early Zero-style example conversation preserved for reference: ```text 👶: 请介绍一下自己。 🤖️: 作为人工智能，我没有实际的生活，也没有自我意识，所以没有自己的生活。我被设计成能够帮助用户解答问题、提供信息、进行对话等。我的设计和功能是由计算机科学、机器学习、人工智能技术和算法所构建的，我通过编程和算法实现。 👶: 推荐一些杭州的特色美食吧。 🤖️: 杭州有很多美味的特色美食，比如鸡肉串、鳗鱼头、水煮鱼、豆腐脑等。这些美食都非常美味，有着独特的风味和口感，绝对是值得一试的美食。 👶: 请为我讲解"大语言模型"这个概念。 🤖️: 大语言模型是一种人工智能技术，它使用大量的文本数据来训练模型，然后在不断的迭代中不断优化模型。它的目标是根据输入的数据来生成符合特定需求的文本，这些文本可以是任何你感兴趣的主题。 👶: Introduce the history of the United States, please. 🤖️: 您提到的"Introok's the believeations of theument." 这个名字来源于中国古代的"groty of of the change." ``` Although this version already has basic dialogue capability, its factual knowledge and generalization ability are still limited; it is mainly an early reference for the feasibility of the Zero training route. Zero model weights are saved as `full_sft_zero_768.pth` (see MiniMind model file links below); if interested, you can download and experience its dialogue performance. --- ## Ⅱ Main Training (Required) > All training scripts are executed from the `cd ./trainer` directory ### 1' Pretraining (Pretrain): An LLM first needs to absorb foundational knowledge and language patterns into its parameters. Only after this stage is sufficiently stable can the model begin to understand questions, organize responses, and develop usable generation ability. Pretraining essentially exposes the model to large amounts of text, such as Wikipedia, news, books, and dialogue corpora, so that it can learn factual knowledge, language patterns, and statistical relationships between contexts. This stage is usually "unsupervised": humans do not label each line as right or wrong; instead, the model extracts patterns from massive text and gradually builds internal representations of world knowledge and language structure. In plain terms, the core objective at this stage is **high-quality next-token continuation**. For example, given the input "秦始皇" (Qin Shi Huang), the model should be able to continue with "是中国历史上的第一位皇帝" (was the first emperor in Chinese history) — content that is semantically and factually consistent. ```bash # Method 1 torchrun --nproc_per_node 1 train_pretrain.py # 1 means single GPU training, adjust according to your hardware (set >=2) # Method 2 python train_pretrain.py ``` > The trained model weight files are saved by default every `save_interval steps` as: `pretrain_*.pth` (* is the specific model dimension, each save overwrites the previous file) > Loss curve during the pretraining stage with `768dim` configuration ```bash # Simple test on pretraining results: python eval_llm.py --weight pretrain 💬: 为什么天空是蓝色的 🧠: 天空之所以看起来是蓝色的，主要是因为太阳光进入大气层后，短波长的蓝光更容易被空气分子散射，因此人眼从各个方向接收到的蓝光会更多。 💬: 解释什么是机器学习 🧠: 机器学习是人工智能的一个重要分支，它通过数据训练模型，使系统能够自动学习规律，并在分类、预测、推荐、自然语言处理等任务中持续改进效果。 ``` ### 2' Supervised Fine-Tuning (SFT): SFT is not only about making the model "chat better"; it can also continue to inject new knowledge, behavioral patterns, and response styles into the model. With `14GB` of SFT data in the current MiniMind main branch, this stage is already more than simple format alignment and is closer to a continued mid-training process. If pretraining lets the model read broadly and acquire basic language ability, then SFT performs further processing on higher-quality and more targeted data. It helps the model adapt to multi-turn dialogue, Q&A, tool calling, and thinking-tag interaction formats, while also imprinting specific knowledge distributions, task patterns, and assistant styles into the parameters. Specifically in MiniMind, the SFT stage lets the model adapt to the multi-turn dialogue template used by the current repository. The model gradually understands the role structure of `user / assistant / system / tool`, while further strengthening instruction following, stable responses, and task completion capabilities. The current training applies truncation to instruction and response lengths, mainly to balance VRAM usage and training efficiency. If longer contexts are needed later, a small number of long-context samples can be used for incremental fine-tuning. During inference, YaRN extrapolation can extend the context length to 2048 and beyond without additional training. ```bash # Method 1 torchrun --nproc_per_node 1 train_full_sft.py # Method 2 python train_full_sft.py ``` > The trained model weight files are saved by default every `save_interval steps` as: `full_sft_*.pth` (* > is the specific model dimension, each save overwrites the previous file) > Loss curve during the SFT stage with `768dim` configuration ```bash # Simple test on SFT results: python eval_llm.py --weight full_sft 💬: 解释什么是机器学习 🧠: 机器学习是人工智能的核心技术之一，通过算法让计算机从数据中学习规律，并持续改进预测或决策效果，常见应用包括推荐系统、图像识别、语音识别和自然语言处理。 💬: 推荐一些中国的美食 🧠: 例如北京烤鸭、兰州拉面、四川火锅、广东早茶、小笼包和麻婆豆腐等，这些美食分别代表了不同地区的风味特点，也很适合作为了解中国饮食文化的入门选择。 ``` ## Ⅲ Other Training (Optional) > All training scripts are executed from the `cd ./trainer` directory ### 3' Knowledge Distillation (KD) Knowledge distillation can be broadly divided into black-box and white-box categories. The current MiniMind main branch involves both approaches, with different emphases. * Black-box distillation: More common, and more aligned with the current main branch's actual practice. Strictly speaking, it is essentially supervised fine-tuning oriented towards teacher outputs, i.e., continuing to train based on hard labels; as LLMs became popular, this approach of "doing FT against strong model outputs" has gradually been broadly categorized under the distillation umbrella, hence commonly called black-box distillation. It focuses on learning answers, styles, and behavioral patterns — the student model can only see "what the teacher said" but cannot see how the teacher internally arrived at that judgment. High-quality answers from `DeepSeek R1`, `Qwen3`, as well as `tool call`, `reasoning`, chain-of-thought data, etc., can all be seen as black-box distillation signals; the current main branch `full_sft` data in MiniMind already has a considerable portion of this approach mixed in. * White-box distillation: Goes further, not only learning the teacher's final outputs but also learning the teacher's preferences at the token distribution level. Compared to black-box distillation, it additionally leverages the finer-grained distribution information from the teacher model's output layer, so the student model learns not just the "standard answer" but also the teacher's relative preferences among candidate tokens. Corresponding to `train_distillation.py`, the current implementation continues training the student model with distribution signals provided by the teacher model on top of already SFT-completed weights, making it more suitable as a reference implementation for understanding MiniMind's distillation pipeline. Black-box distillation is essentially equivalent to supervised fine-tuning on teacher-generated answers: ```math \mathcal{L}_{blackbox} = \mathrm{CE}(y_{teacher}, p_{student}) ``` White-box distillation typically fits the teacher distribution in addition to the supervised loss: ```math \mathcal{L}_{whitebox} = \alpha \mathcal{L}_{CE} + (1-\alpha) T^2 \mathrm{KL}(p_t^T \parallel p_s^T) ``` The `train_distillation.py` script is intended as a reference implementation for understanding the white-box distillation pipeline: it demonstrates teacher/student dual model loading, `CE + KL` mixed loss, temperature scaling, MoE and dense combination distillation, as well as key details such as checkpoint resume and distributed training. ```bash # Method 1 torchrun --nproc_per_node 1 train_distillation.py # Method 2 python train_distillation.py ``` ### 4' LoRA (Low-Rank Adaptation) LoRA is a common Parameter-Efficient Fine-Tuning (PEFT) method. Compared with full-parameter fine-tuning, it only updates a small number of newly added parameters while keeping the original model weights unchanged, reducing training cost and making it better suited to vertical-domain adaptation. Its core idea is to introduce low-rank incremental branches alongside the original weight matrices, training only these low-rank parameters, thereby completing capability transfer with relatively small cost. Related implementations can be found in `model_lora.py` and `train_lora.py`; the entire pipeline is purely hand-written, without relying on third-party wrappers. ```bash # train_lora.py can usually be completed fairly quickly even on CPU # Method 1 cd trainer && torchrun --nproc_per_node 1 train_lora.py # Method 2 cd trainer && python train_lora.py ``` > The trained model weight files are saved by default every `save_interval steps` as: `lora_xxx_*.pth` (* is the specific model dimension, each save overwrites the previous file) LoRA is well-suited for handling problems like "how to let the model quickly adapt to private domains or vertical scenarios while preserving general capabilities as much as possible." For example, when the base model lacks medical knowledge, a medical-oriented LoRA weight layer can be stacked on top of the original model to achieve better domain performance at relatively small cost. Usually you only need to prepare multi-turn dialogue format data in the same way, place it in `lora_xxx.jsonl`, and then run `cd trainer && python train_lora.py` from the repository root to obtain new `LoRA` model weights. Example 1: Vertical domain data ```jsonl {"conversations": [{"role": "user", "content": "请问颈椎病的人枕头多高才最好？"}, {"role": "assistant", "content": "颈椎病患者选择枕头的高度应该根据..."}]} {"conversations": [{"role": "user", "content": "请问xxx"}, {"role": "assistant", "content": "xxx..."}]} ``` Example 2: Self-awareness data ```jsonl {"conversations": [{"role": "user", "content": "你叫什么名字？"}, {"role": "assistant", "content": "您好，我名叫 MiniMind，是由 Jingyao Gong 开发的人工智能助手。我可以帮助你解决各种问题，包括翻译、推荐内容等等。"}]} {"conversations": [{"role": "user", "content": "你是谁"}, {"role": "assistant", "content": "您好，我是 MiniMind，由 Jingyao Gong 创建。我的主要职责是提供准确的回答和帮助。"}]} ``` At this point, the [base model + LoRA weights] can be used in combination, preserving the base model's general capabilities while enhancing specific domain performance. After training, simple tests can also be done via `eval_llm.py`. ```bash # Note: the weight parameter specifies the base model type, which must be consistent with the base model used during train_lora training python eval_llm.py --weight full_sft --lora_weight lora_medical 👶: 我最近经常感到头晕，可能是什么原因？ 🤖️: 头晕可能是由于多种原因，包括但不限于：心脏疾病、眼睛损伤、过敏反应、皮肤问题、过敏反应、压力、焦虑、疲劳、药物副作用、性功能障碍或者其他... python eval_llm.py --weight full_sft --lora_weight lora_identity 👶: 你和openai是什么关系？ 🤖️: 我是 MiniMind，一个由 Jingyao Gong 开发的人工智能助手。我通过自然语言处理和算法训练来与用户进行交互。 ``` PS: If you have sufficient data, you can also perform `full_sft` full-parameter fine-tuning directly. This usually requires careful mixing of general and domain-specific data; otherwise, the model may lose part of its general capability due to overfitting on vertical-domain samples. > `LoRA` weights can be merged back into the base model and exported as new complete model weights, using `convert_merge_base_lora` in `scripts/convert_model.py`: ```bash cd scripts && python convert_model.py ``` ### **5' Tool Calling & Adaptive Thinking** Starting from `2026-03`, the repository removed the standalone `train_reason.py`. The current version no longer separately maintains `reason_*.pth` weights, but instead uniformly models "whether to explicitly output the thinking process" through `chat_template`, `` tags, `open_thinking` switch, and subsequent SFT / RLAIF pipelines. #### 5.1 Tool Calling The current `toolcall` capability has been merged into the `sft_t2t` / `sft_t2t_mini` main branch data, so an additional Tool Calling SFT stage is usually unnecessary; the default `full_sft` weights already have basic Tool Call capability. The current training data for this part mainly contains about `100K` samples generated from `qwen3-4b`, and the tool list covers around `10` simulated custom tools, such as time queries, math calculation, and weather lookup. At this stage, its generalization ability is still limited. Tool Calling samples follow the OpenAI-style multi-turn message format: ```jsonl { "conversations": [ {"role": "system", "content": "# Tools ...", "tools": "[...]"}, {"role": "user", "content": "帮我算一下 256 乘以 37 等于多少"}, {"role": "assistant", "content": "", "tool_calls": "[{\"name\":\"calculate_math\",\"arguments\":{\"expression\":\"256 * 37\"}}]"}, {"role": "tool", "content": "{\"result\":\"9472\"}"}, {"role": "assistant", "content": "256 乘以 37 等于 9472。"} ] } ``` Here, `tools` is attached to the `system` message, and `tool_calls` is attached to the `assistant` message. During training, the `chat_template` automatically expands them into `...` and `...` segments, allowing the model to learn the native tool-call format directly. Tool Calling's chat template has been unified to parse as: ```text {"name": "...", "arguments": {...}} {...tool result...} ``` You can also directly run simple tests via `eval_toolcall.py`: ```bash python eval_toolcall.py --weight full_sft 💬: 现在几点了？ 🧠: {"name": "get_current_time", "arguments": {"timezone": "Asia/Shanghai"}} 📞 [Tool Calling]: get_current_time ✅ [Tool Called]: {"datetime": "2026-03-15 17:18:22", "timezone": "Asia/Shanghai"} 🧠: 现在是2026年3月15日17时18分22秒。 ``` #### 5.2 Adaptive Thinking `minimind` unifies explicit thinking capability at the template level, which is also consistent with the template design of many current mainstream large models: - `open_thinking=0`: By default injects an empty `\n\n`, the model tends to answer directly; - `open_thinking=1`: The template pre-injects the `` start tag, and the model then continues to output the explicit thinking process and final answer; - CLI, OpenAI-API, and WebUI all support this switch. More precisely, the approach is no longer to train a separate thinking model, but to push "whether to think explicitly" down to the `chat_template`. The template layer reserves the `` structure, and the same model switches dynamically through `open_thinking` during inference. During training, empty `think`, explicit `reasoning_content`, and `thinking_ratio` sampling are mixed so that the model gradually learns when to think explicitly and when to answer directly. ```bash # Test responses python eval_llm.py --load_from ./minimind-3 --open_thinking 1 ``` OpenAI-API-SDK usage: ```python response = client.chat.completions.create( model="minimind", messages=[{"role": "user", "content": "你是谁？"}], # ... extra_body={"chat_template_kwargs": {"open_thinking": True}} # Thinking switch ) ``` Note: When Tool Call and explicit thinking are enabled simultaneously, the model usually cannot stably output the thinking process. The reason is that the current training data still lacks joint distillation samples where "reasoning and tool call coexist", so the model has not yet fully learned the coordinated expression of these two capabilities. ## Ⅳ Reinforcement Learning (Optional) In the post-training practice of LLMs, there are mainly two common reinforcement learning paths: 1. **Reinforcement Learning from Human Feedback (RLHF)** - Trains the model by evaluating model outputs through **human** preference assessments, making it generate content that better aligns with human values and preferences. 2. **Reinforcement Learning from AI Feedback (RLAIF)** - Uses **AI models** or other automatically verifiable mechanisms to provide feedback, without directly relying on human annotation. - Here, "AI feedback" in a broad sense can also extend to rule rewards, Ground Truth verification, code interpreters, environment feedback, and other automated signals. | Type | Judge | Advantages | Disadvantages | |-------|-------|------------|---------------| | RLHF | Human | Closer to real human preferences | High cost, low efficiency | | RLAIF | Model | Automated, highly scalable | May deviate from real human preferences | Both essentially belong to the reinforcement learning paradigm of optimizing model behavior using some form of "**feedback**". However, in specific practice, their differences go beyond just the feedback source: whether the reward is verifiable, whether it is continuous, whether it depends on environment interaction, and whether it is delayed until the end of the entire episode — all directly affect the training form and engineering implementation. ### 👀 A Unified Perspective on PO Algorithms Before introducing the implementation of specific algorithms, let me first describe the unified commonality of all Policy Optimization (PO) algorithms from my own minimalist perspective. The essence of all RL algorithms is just optimizing an expectation: $$\mathcal{J}_{PO} = \mathbb{E}_{q \sim P(Q), o \sim \pi(O|q)} \left[ \underbrace{f(r_t)}_{\text{policy term}} \cdot \underbrace{g(A_t)}_{\text{advantage term}} - \underbrace{h(\text{KL}_t)}_{\text{regularization term}} \right]$$ During training, one only needs to **minimize the negative objective function**, i.e.: $\mathcal{L}_{PO} = -\mathcal{J}_{PO}$ This framework contains only three core components: * **Policy term** $f(r_t)$: How to use the probability ratio $r_t$? It tells the model how far the new and old policies have deviated, and whether better tokens have been explored * **Advantage term** $g(A_t)$: How to compute the advantage $A_t$, this is very important! It's no surprise that large models can solve definite integrals correctly, but for small models, even getting addition and subtraction right usually yields a positive advantage * **Regularization term** $h(\text{KL}_t)$: How to constrain the magnitude of change $\text{KL}_t$, preventing both drifting too far and constraining too tightly (Expand) Notation Guide | Symbol | Meaning | Description | Range | |--------|---------|-------------|-------| | $q$ | Question/Prompt | Sampled from dataset $P(Q)$ | - | | $o$ | Model output sequence | Generated by policy $\pi$ | - | | $r_t$ | Probability ratio | $r_t = \frac{\pi_\theta(o_t \mid q, o_{ The trained model weight files are saved by default every `save_interval steps` as: `dpo_*.pth` (* is the specific model dimension, each save overwrites the previous file) ### 7' Reinforcement Learning from AI Feedback (RLAIF) A small naming caveat is needed here. I still refer to this section as `RLAIF`, although the term is not strictly precise. Routes such as RLVR, which rely on verifiable rewards, have their own lineage and do not fit neatly into the narrow definition of AI feedback. If "AI" is interpreted more broadly, however, the name is still defensible: rewards may come from reward models, judge models, or other explicit intelligent agents, but they may also come from rule functions, Ground Truth verification, tool-call results, environment states, and other automatically available signals. When the rules are complex enough and the symbolic system is rich enough, the boundary between these signals and "intelligent feedback" is not always clear. This chapter therefore focuses on reinforcement learning after SFT using various **non-human, automatically obtainable feedback signals**. For example, whether a math answer is correct, whether generated code passes test cases, or whether the reasoning process follows the expected format can all be judged automatically. For single-turn verifiable tasks, this feedback is often closer to an immediate reward. In Agentic RL scenarios, rewards are more often delayed until the end of a multi-step interaction, or come directly from the environment itself. Their common characteristics are usually **on-policy training** and **high scalability**: no expensive human annotation is required, and large numbers of training samples can be generated for online trial and error. MiniMind has implemented **2+N** basic + cutting-edge RLAIF methods: * **PPO**, **GRPO** — classic RL algorithms validated at large scale * N cutting-edge RL algorithms (updated periodically on an experimental basis) **1️⃣ Dataset Preparation (Required)** The current main branch uses `rlaif.jsonl` as the RLAIF training data. It is approximately `20MB` in size, more complete than the earlier `rlaif-mini.jsonl`, and better suited for directly verifying the training behavior of PPO / GRPO / CISPO. The data format is consistent with SFT, but the assistant does not need content, because during training it is entirely generated in real-time by the $\Pi$ policy model through sampling. Therefore it looks like: ```json { "conversations": [ {"role": "user", "content": "请解释一下什么是光合作用？"}, {"role": "assistant", "content": "无"} ] } ``` During the RLAIF training process, the model generates 1 or more candidate responses based on the user's question, and then a reward function/model scores the responses. High-scoring responses will be encouraged (increasing the $\Pi$ policy probability), and low-scoring responses will be suppressed (decreasing the $\Pi$ policy probability). This "score -> adjust" loop is the core of reinforcement learning. **2️⃣ Reward Mechanism Preparation (Required)** RLAIF training requires some form of computable reward signal; it can come from a reward model, or from rule functions, Ground Truth verification, or environment feedback. MiniMind currently demonstrates the Reward Model route by default. Here we select the small and high-quality `InternLM2-1.8B-Reward` ([ModelScope](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm2-1_8b-reward) | [HuggingFace](https://huggingface.co/internlm/internlm2-1_8b-reward)) as the base reward model. After downloading the reward model, it needs to be placed in the **sibling directory** of the minimind project, with the recommended structure as follows: ``` root/ ├── minimind/ # MiniMind project │ ├── model/ │ └── ... └── internlm2-1_8b-reward/ # Reward model ├── config.json ├── model.safetensors └── ... ``` **Reward Mechanism Selection & MiniMind Limitations (Click to Expand)** **1. Diversity of Reward Mechanisms** The "reward signal" sources in RLAIF can be very flexible: - **Model-based rewards**: Can use a dedicated Reward Model (such as InternLM2-Reward), or use a general LLM + prompts for scoring (such as Qwen3-as-a-Judge). The scale and architecture of the reward model can be freely chosen. - **Rule-based rewards**: Reward signals can be constructed based on rule functions, for example: - Math problem answer correctness verification (Ground Truth comparison) - SQL execution success rate and result accuracy - Code interpreter execution results (pass@k) - Tool call return status (API success/failure) - Format compliance checking (JSON/XML parsing) - Reasoning chain completeness evaluation (number of CoT steps) - **Environment-based rewards**: In Agent scenarios, environment feedback itself serves as a natural reward (such as game scores, research completeness, task completion rate). Any mechanism that can quantify "response quality" can serve as an RL reward source. DeepSeek R1 is a typical example: using rule functions to verify math answer correctness as rewards, without needing an additional Reward Model. **2. MiniMind Limitation: Reward Sparsity Problem** RLAIF training can target both reasoning models and non-reasoning models; the difference is only in format. However, for models like MiniMind with extremely small 0.1B parameters and weak capabilities, serious Reward Sparsity problems will be encountered on general tasks (such as R1-style math datasets): - **Phenomenon**: Almost all candidate responses generated by the model are incorrect, resulting in all reward scores $r(x,y) \approx 0$ - **Consequence**: The advantage function $A(x,y) = r(x,y) - b(x) \approx 0$, the policy gradient signal vanishes, and parameters $\theta$ cannot be effectively updated It's like having an elementary school student take college entrance exam math problems — no matter how many attempts, they always score zero, unable to learn improvement strategies from score differences. Therefore, this is a fundamental limitation of the RL algorithm's principles. To mitigate this problem, MiniMind's implementation chose **model-based continuous reward signals**: - The Reward Model outputs continuous scores (e.g., -2.5 to +3.0), rather than binary 0/1 - Even when all response quality is poor, it can still distinguish subtle differences between "even worse" (-3.0) and "worse" (-2.8). So this kind of **dense and continuous** reward signal can provide non-zero gradients for the advantage function $A(x,y)$, enabling the policy network to optimize incrementally - Multiple reward sources can also be mixed: $r_{\text{total}} = \alpha \cdot r_{\text{model}} + \beta \cdot r_{\text{rule}}$ (for example, both detecting think tag format reward and combining the reward score for the response quality itself) - In MiniMind practice, avoid directly using rule-based binary rewards + difficulty beyond capability (such as MATH500), which easily leads to all-zero rewards; - Monitor training by observing the variance of reward scores $\text{Var}(r)$; if it stays close to 0, the data or reward mechanism needs to be adjusted **For production-level large model Agentic RL scenarios**: In real Agent systems (code generation, tool calling, multi-turn chains of retrieval-planning-execution), rewards follow a different paradigm of "delayed settlement over the entire episode": - The LLM needs to generate tool call instructions (tool_call) token by token, go through parsing (tool_parse), tool execution (tool_exec), then splice the results back into the context to continue the next step; repeating until completion. - A complete task chain includes multiple calls + thinking, until the termination condition is met and a total reward is calculated once (such as whether the task is completed, whether tests pass, whether the target is hit). Therefore, Agentic RL is closer to the sparse/delayed reward setting: gradient backpropagation only occurs "after the entire episode ends", which is very different from non-Agentic RL tasks that "score instantly and update instantly" on a single dialogue turn. This also explains why Agent tasks lean more towards environment-based reward, rather than static scoring by Reward Models. - **Environment interaction feedback**: Ultimately based on execution results (whether code runs successfully, whether API returns success, whether sub-goals are completed); - **Model-based reward limitations**: Limited in capturing the full picture of long-chain, executable semantics, and highly likely to be inconsistent with real environment feedback (reward hacking). --- #### 7.1 [Proximal Policy Optimization](https://arxiv.org/abs/1707.06347) PPO is a very classic reinforcement learning algorithm proposed by OpenAI in 2017, and is also one of the most common baseline methods in the LLM RL field. **PPO Loss**: $$\mathcal{L}_{PPO} = -\mathbb{E}\left[\min(r_t \cdot A_t, \text{clip}(r_t, 1-\varepsilon, 1+\varepsilon) \cdot A_t)\right] + \beta \cdot \mathbb{E}[\text{KL}]$$ Where: - **Policy term**: $f(r_t) = \min(r_t, \text{clip}(r_t, 1-\varepsilon, 1+\varepsilon))$ (clips probability ratio to prevent overly aggressive updates) - **Advantage term**: $g(A_t) = R - V(s)$ (estimates value function through Critic network) - **Regularization term**: $h(\text{KL}_t) = \beta \cdot \mathbb{E}[\text{KL}]$ (global KL divergence constraint) Compared to DPO, - DPO (Off-Policy): Training data consists of static preference pairs (chosen vs rejected), which can be reused across multiple training epochs, like traditional supervised learning. High data efficiency, low cost, and no Reward Model needed. - PPO (On-Policy): Must use the current policy to sample new data in real-time; old policy data can only be reused to a limited extent, otherwise distribution shift will occur. Although importance sampling and clip allow slight deviation, it fundamentally still requires data from a relatively recent policy. Lower data efficiency, but more suitable for exploratory learning. Simply put: - The former learns according to offline predetermined "good/bad standards"; - The latter samples online based on the latest policy and corrects in real-time. MiniMind's PPO implementation includes Actor (generating responses), Critic (evaluating response value), and complete GAE (Generalized Advantage Estimation) advantage function computation. **Training method**: ```bash # Method 1 torchrun --nproc_per_node N train_ppo.py # Method 2 python train_ppo.py ``` > The trained model weight files are saved by default every `save_interval steps` as: `ppo_actor_*.pth` (* is the specific model dimension) > MiniMind optimization trends during the PPO training stage From the training curves, it can be seen that PPO has the problem of **slow reward improvement**. I personally believe this mainly stems from PPO's **dual-network joint optimization** approach: the Critic needs to gradually converge to accurately estimate the value function, while the Actor's policy updates depend on the advantage estimates provided by the Critic. The two are interdependent, forming a complex optimization process. In the early stages of training, inaccurate Critic estimates affect the Actor's gradient direction, leading to overall slow convergence. Additionally, PPO needs to maintain two networks simultaneously, and under the current implementation, VRAM usage is approximately 1.5–2 times that of single-network methods. #### 7.2 [Group Relative Policy Optimization](https://arxiv.org/pdf/2402.03300) In early 2025, as DeepSeek-R1 exploded in popularity, GRPO from the DeepSeekMath paper also quickly entered the mainstream spotlight, becoming one of the most watched RL algorithms for a time. However, the AI field has always iterated extremely fast. As of today, GRPO has evolved more into a common baseline for various XXPO variants (such as DAPO, GSPO, CISPO, etc.). Its core innovation can be summarized in one sentence: "group relative value estimation." **GRPO Loss**: $$\mathcal{L}_{GRPO} = -\mathbb{E}\left[\min(r_t \cdot A_t, \mathrm{clip}(r_t, 1-\varepsilon, 1+\varepsilon) \cdot A_t) - \beta \cdot \text{KL}_t\right]$$ Where: - **Policy term**: $f(r_t) = \min(r_t, \mathrm{clip}(r_t, 1-\varepsilon, 1+\varepsilon))$ (uses symmetric clip on the probability ratio) - **Advantage term**: $g(A_t) = \frac{R - \mu_{group}}{\sigma_{group}}$ (intra-group normalization, eliminating the Critic network) - **Regularization term**: $h(\text{KL}_t) = \beta \cdot \text{KL}_t$ (token-level KL divergence constraint) For the same question, the model generates N responses and computes their respective rewards, then uses the intra-group average reward as a baseline. Responses above the baseline are encouraged, and responses below the baseline are suppressed, thus no additional critic network needs to be trained. A more prominent issue with GRPO is Degenerate Groups: if for a certain question the rewards of N responses are almost identical, then the learning signal for this group will be close to 0. On ultra-small models like MiniMind, this problem is especially pronounced, so training must be constrained within reasonable capability boundaries. **Training method**: ```bash # Method 1 torchrun --nproc_per_node N train_grpo.py # Method 2 python train_grpo.py ``` > The trained model weight files are saved by default every `save_interval steps` as: `grpo_*.pth` > MiniMind optimization trends during the GRPO training stage From the training curves, it can be seen that GRPO's **reward shows a more stable upward trend**, reaching around 4, indicating that GRPO itself can better utilize RLAIF signals. Policy Loss decreases steadily overall, and compared to PPO's dual-network optimization, GRPO's single-network architecture trains more stably with a higher convergence ceiling. #### 7.3 [Clipped Importance Sampling Policy Optimization](https://huggingface.co/papers/2506.13585) Among the many XXPO variants, I personally found CISPO particularly memorable. It does not reinvent an entire complex framework; instead, it directly targets a long-standing awkward problem in PPO/GRPO: once the ratio is clipped, the gradient flow can be hard-truncated. CISPO does not focus on redesigning the group baseline. Instead, it uses a very small loss modification to fix this problem more directly. **CISPO Loss**: $$\mathcal{L}_{CISPO} = -\mathbb{E}\left[\min(r_t, \varepsilon_{max}) \cdot A_t \cdot \log \pi_\theta(a_t|s) - \beta \cdot \text{KL}_t\right]$$ Where: - **Policy term**: $f(r_t) = \min(r_t, \varepsilon_{max}) \cdot \log \pi_\theta(a_t|s)$ (ratio serves only as a clipped weight) - **Advantage term**: $g(A_t) = \frac{R - \mu_{group}}{\sigma_{group}}$ (can directly reuse GRPO's intra-group relative advantage) - **Regularization term**: $h(\text{KL}_t) = \beta \cdot \text{KL}_t$ (token-level KL divergence constraint) CISPO, building on GRPO, rewrites the policy term that was easily clipped into a constant into the form "clipped weight × log probability". This way, even if the ratio is truncated, the gradient path is not truncated along with it. Therefore, CISPO can be directly viewed as a loss variant of GRPO to implement, rather than maintaining a separate standalone script. No separate experiment is listed here. One only needs to set `loss_type` to `cispo` in `train_grpo.py`; the rest of the training process still follows GRPO's group sampling, reward computation, and advantage construction logic. #### 7.4 Agentic RL 🔥 The concept of "Agentic" is broad, and here it is used in a narrower sense: the goal is to let small models like MiniMind (~100M) learn basic calling, observation, and re-planning capabilities on a limited tool set, rather than covering the full scope of state management, long-term memory, and complex workflow orchestration in a complete Agent system. Starting from `2026-03`, the repository added `train_agent`, which begins to support a form of multi-turn Tool-Use RL closer to real interaction. This is a training script I personally enjoy a lot: it combines RLVR / RLAIF-style data organization with online RL rollout, went through many iterations of debugging, and encountered issues such as convergence failure, reward hacking, and multi-turn context misalignment, but ultimately still preserves MiniMind's consistent simplicity and readability. The data for this part is `agent_rl.jsonl` / `agent_rl_math.jsonl`. Compared to regular dialogue data, they have an additional `gt` as the final verification target; if we denote a sample as $(x, \mathcal{T}, gt)$, then the optimization target during training is no longer a single-turn response $y$, but a multi-turn trajectory $\tau$: $$ \tau = (a_1, o_1, a_2, o_2, \dots, a_T), \quad a_t \sim \pi_\theta(\cdot \mid s_t, \mathcal{T}) $$ Here, `chat_template` uniformly organizes `tools / tool_calls / tool` messages. If a step generates a `tool_call`, the tool is executed, the observation is appended back to the context, and rollout continues. The main pipeline can be compressed to: $$ \texttt{rollout batch} \rightarrow \texttt{calculate rewards} \rightarrow \texttt{policy update} $$ The reward is also jointly scored on the entire trajectory: $$ R(\tau) = R_{\text{answer}} + R_{\text{tool}} + R_{\text{format}} + R_{\text{rm}} - R_{\text{unfinished}} $$ Here, tool call legality, `gt` hits, format closure, unfinished penalty, and Reward Model scores are all considered simultaneously. Compared to regular PPO / GRPO, this involves multi-turn rollout and delayed reward. **Training method**: ```bash # ① Default: use torch for rollout # Method 1 torchrun --nproc_per_node N train_agent.py # Method 2 python train_agent.py ``` ```bash # ② Use sglang for rollout # Start sglang server first: python -m sglang.launch_server --model-path ./minimind-3 --attention-backend triton --host 0.0.0.0 --port 8998 # Training parameters for reference: python train_agent.py --rollout_engine sglang --sglang_base_url http://localhost:8998 --sglang_shared_path ./ckpt_mm --data_path ../dataset/agent_rl_math.jsonl --use_wandb ``` > The trained model weight files are saved by default every `save_interval steps` as: `agent_*.pth` > MiniMind optimization trends during the Agentic RL training stage Here I'll also briefly mention the `rollout_engine`. The so-called "training-inference separation" means decoupling **parameter updates** and **trajectory rollout**: the training side handles policy optimization, while the rollout side handles high-throughput sampling. From the top level, they uniformly present as "give me a prompt, I'll return rollout results; after training is done, sync the new weights back." Therefore, the training script doesn't need to care whether the underlying implementation is local `generate` or a remote `inference` engine. Note that the current implementation is still **synchronous** (sample a batch, then update), not yet asynchronous training with a pure rollout buffer. > Schematic diagram of the decoupled RL structure in MiniMind: training side, trajectory side, and rollout side If we draw an analogy to larger-scale systems, it already has the flavor of large-scale RL frameworks like openrlhf/verl/slime: - The left side is the training side, responsible for policy updates - The right side is the rollout / inference side, responsible for throughput sampling - The middle connects through trajectory and weight synchronization - Tool execution and environment feedback do not directly enter the loss, but directly affect the reward quality of the entire trajectory I personally view this implementation as a very interesting transitional version within MiniMind. Although it is still far from an industrial-grade Agent training framework, it has already connected the key elements end to end: **template organization, tool execution, multi-turn rollout, delayed reward, and training-inference separation**. Perhaps there is nothing simpler than it for now. ```bash # Test the final model's Tool Use capability python eval_toolcall.py --weight agent 💬: 现在几点了？ 🧠: {"name": "get_current_time", "arguments": {"timezone": "Asia/Shanghai"}} 📞 [Tool Calling]: get_current_time ✅ [Tool Called]: {"datetime": "2026-03-15 21:22:33", "timezone": "Asia/Shanghai"} 🧠: 现在是2026年3月15日21时22分33秒（北京时间）。 💬: 帮我生成一个1到1000的随机数，然后计算它的平方 🧠: {"name": "random_number", "arguments": {"min": 1, "max": 1000}} 📞 [Tool Calling]: random_number ✅ [Tool Called]: {"result": 71} 🧠: {"name": "calculate_math", "arguments": {"expression": "71**2"}} 📞 [Tool Calling]: calculate_math ✅ [Tool Called]: {"result": "5041"} 🧠: 生成的1到1000的随机数是71，根据计算结果，71的平方等于5041。 ``` > Based on AgentRL training results testing, supports thinking display, tool selection, and multi-turn Tool Use interaction ### 🖊️ RL Summary Returning to the "**unified framework**", the table below summarizes how different PO algorithms instantiate the same three core components: | Algorithm | Policy term $f(r_t)$ | Advantage term $g(A_t)$ | Regularization term $h(\text{KL}_t)$ | Number of training models | |-----------|---------------------|------------------------|-------------------------------------|--------------------------| | **DPO** | $\log r_w - \log r_l$ | No explicit advantage term | Implicit in $\beta$ | 1 (2 participate in forward) | | **PPO** | $\min(r, \text{clip}(r))$ | $R - V(s)$ | $\beta \cdot \mathbb{E}[\text{KL}]$ | 2 | | **GRPO** | $\min(r, \text{clip}(r))$ | $\frac{R - \mu}{\sigma}$ | $\beta \cdot \text{KL}_t$ | 1 | | **CISPO** | $\mathrm{clip}(r, 0, \varepsilon_{max}) \cdot A_t \cdot \log \pi_\theta$ | $\frac{R - \mu}{\sigma}$ | $\beta \cdot \text{KL}_t$ | 1 | **To put it plainly, these RL algorithms are not isolated from one another. From a unified optimization perspective, they are natural variants formed by making different design trade-offs on the same objective function, presenting a beautifully self-consistent unity.** ## Ⅴ Open-Sourced Training Results 📦 #### ① PyTorch Models ([ModelScope](https://www.modelscope.cn/models/gongjy/minimind-3-pytorch) | [HuggingFace](https://huggingface.co/jingyaogong/minimind-3-pytorch)) > Note: Model weights are subject to actual releases. Not all training stages or experimental branches (such as DPO, PPO, GRPO, CISPO, Agent, LoRA, etc.) will be continuously maintained and separately published; some weights are only used for experimental verification or learning purposes. As data iterates or models are adjusted, the necessity of synchronizing all versions one by one is limited and would incur high maintenance and training costs. Torch File Naming Reference - Dense: - Pretrain: `pretrain_{hidden_size}.pth` - SFT: `full_sft_{hidden_size}.pth` - DPO: `dpo_{hidden_size}.pth` - PPO: `ppo_actor_{hidden_size}.pth` - GRPO: `grpo_{hidden_size}.pth` - Agent: `agent_{hidden_size}.pth` - LoRA: `lora_xxx_{hidden_size}.pth` - MoE: - The corresponding weights with the same name have a `_moe` suffix appended at the end, for example: `pretrain_{hidden_size}_moe.pth`, `full_sft_{hidden_size}_moe.pth` #### ② Transformers Models ([ModelScope](https://www.modelscope.cn/collections/MiniMind-b72f4cfeb74b47) | [HuggingFace](https://huggingface.co/collections/jingyaogong/minimind-66caf8d999f5c7fa64f399e5)) > Note: Unless otherwise specified, `transformers` versions are usually converted from `full_sft` weights. RL-type post-training tends to optimize around specific reward targets; although it usually improves the reward score, it sacrifices some general capabilities and knowledge; this kind of reward hacking / capability trade-off is difficult to avoid across all models, differing more in degree. --- # 📌 Evaluation ## Ⅰ RL Model Comparison Subjective/objective comparison based on `minimind-3 (64M)` under the same random seed and other hyperparameters, for reference: [A] minimind-3 (64M, SFT) [B] minimind-3 (64M, GRPO) [C] minimind-3 (64M, Agent-CISPO) ### Test 1: Subjective Q&A Comparison ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Test 2: Light Agent Task Comparison A test adapted from the `eval_toolcall` script, using a set of math ToolUse tasks to compare the performance of the current `agent` weights and `full_sft` weights: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### 👉 Comprehensive Evaluation 1 From these results, the current `agent` compared to `full_sft` has already clearly pulled ahead on lightweight Agent tasks with tool calling. Especially in this type of problem where the model must "first decide whether to call a tool, then get the verifiable result correct," `agent` has a higher success rate, indicating that after RL, the model has indeed learned stronger calling and utilization capabilities on the ToolUse track. However, this improvement does not come without a cost. `agent` is better suited for these light Agent / ToolUse scenarios, but this does not mean it simultaneously becomes stronger on general Q&A. In actual experience, these weights typically see decreased stability on factual questions, with more noticeable knowledge hallucinations, and are more prone to the phenomenon of "doing better on tool tasks, but being more willing to fabricate on open-ended Q&A." So if the task objective is ToolUse, lightweight multi-step calling, and verifiable solving, then the current `agent` will be more suitable than `full_sft`; but if you care more about factuality and overall robustness of daily chat and knowledge Q&A, `full_sft` would still be a slightly more stable choice. Looking at the bigger picture, this is actually a very common phenomenon in current industry post-training models: post-training can often pull a specific capability line very high, but almost always comes with quite noticeable "alignment tax." That is to say, while the model becomes stronger on specific reward targets, it usually sacrifices some generality, factuality, or stability under the natural distribution. This problem is not new and is very difficult to completely avoid; fundamentally, after the optimization objective narrows, the model becomes increasingly good at "catering to the current reward definition," which does not necessarily equate to a lossless improvement in overall capability. ## Ⅱ Comparison with Other Models > Note: The following comparison is only for experiential reference, not a strict benchmark; sample size is limited and involves subjectivity. [A] minimind-3 (0.06B) [B] minimind-3-moe (0.2B-A0.06B) [C] [baby-llama2-chinese (0.2B)](https://github.com/DLLXW/baby-llama2-chinese) [D] [chatlm-mini-chinese (0.2B)](https://github.com/charent/ChatLM-mini-Chinese) ### Test 3: Q&A ``` /* Detailed source-code truncated for AI context efficiency. */ ```python def quick_sort(arr): if len(arr) <= 1: return arr pivot = arr[len(arr) // 2] left = [x for x in arr if x < pivot] middle = [x for x in arr if x == pivot] right = [x for x in arr if x > pivot] return quick_sort(left) + middle + quick_sort(right) # 示例 arr = [3, 6, 8, 10, 1, 2, 1] print(quick_sort(arr)) # 输出: [1, 1, 2, 3, 6, 8, 10] ```这段代码首先检查输入数组的长度是否小于等于1。如果是，则直接返回数组。接下来是数组的长度，如果是数组，则直接返回数组。然后是数组中的元素，如果数组长度小于等于1，则直接返回数组。最后是数组中的元素，如果数组包含重复的元素，则不返回任何元素。 [B]: 快速排序是一种高效的排序算法，采用分治策略来将一个序列分为较小和较大的两个子序列，然后递归地排序这两个子序列。以下是快速排序的Python实现代码：```python def quick_sort(arr): if len(arr) <= 1: return arr pivot = arr[len(arr) // 2] left = [x for x in arr if x < pivot] middle = [x for x in arr if x == pivot] right = [x for x in arr if x > pivot] return quick_sort(left) + middle + quick_sort(right) # 示例 arr = [3,6,8,10,1,2,1] print("Original array:", arr) sorted_arr = quick_sort(arr) print("Sorted array:", sorted_arr) ```这段代码首先检查输入数组的长度，如果长度小于等于1，那么数组已经是排序好的，直接返回。然后选择一个基准值（pivot），将数组分为三部分：小于基准值的元素、等于基准值的元素和大于基准值的元素。接着递归地对小于和大于基准值的元素进行快速排序，并将结果合并。 [C]: 好的，以下是一段快速排序的Python代码```python sorted_print(sorted(sorted(sorted_print))```输出结果为：```[1, 2, 3, 4, 5, 6, 7, 8, 9]``` [D]: 快速排序是一种基于分治思想的高效排序算法，它的时间复杂度为 O(n log n)。它的基本思想是通过一趟排序将待排序序列划分为两个子序列，其中一个子序列的所有元素都比另一个子序列的所有元素小，然后对这两个子序列分别递归地进行排序。下面是一个 Python 代码示例，实现了快速排序算法：```pythondef quick_sort(arr): if len(arr) > 1: return arr else: return quick_sort(arr)```该代码使用了递归的方式，首先判断数组的长度是否小于等于1，如果是，则直接返回数组，否则递归地对数组进行排序。在递归过程中，我们首先判断数组的第一个元素是否大于等于1，如果是，则将数组中的元素与第一个元素进行比较，如果不等于1，则将数组中的元素与第二个元素进行比较，以此类推，直到所有元素都被排序。该代码的时间复杂度为 O(n log n)，是一个非常实用的排序算法。 [Q]: 截至2025年2月9日14时31分，电影《哪吒之魔童闹海》累计票房（含预售）突破78.09亿元，超过《星球大战：天行者崛起》的票房成绩，成为首部进入全球票房榜前40的亚洲电影。这一成就不仅标志着中国电影在国际市场的突破，也展示了中国动画电影的巨大潜力和市场吸引力。从这段文字总结20字的摘要给我。 [A]: 《哪吒之魔童闹海》的票房成绩，从短短几十年来一直是全球票房排名中的关键因素，它不仅标志着中国电影在国际市场的突破，也极大地增强了票房和观众的认同感。以下是对这一成就的简要总结：1. **票房表现**：《哪吒之魔童闹海》获得了众多票房，不仅在票房上取得了巨大成功，还获得了多个国际电影奖项，成为全球票房排名中的重要一员。2. **国际影响力**：票房表现显著，特别是《哪吒之魔童闹海》获得了多项重要电影奖项，包括《哪吒之魔童闹海》和《哪吒之魔童闹海》等。这些电影不仅在国际上获得了巨大成功，也吸引了全球观众的关注。3. **电影艺术价值**：电影不仅展示了中国电影的艺术魅力，还赋予了电影艺术家强大的文化价值和情感深度。《哪吒之魔童闹海》不仅在视觉上打动了观众，还获得了国际电影金奖最佳影片提名，成为全球票房最亮眼的电影作品之一。4. **市场地位**：《哪吒之魔童闘海》获得了众多国际电影奖项，不仅在国内获得了巨大成功，还吸引了来自世界各地的观众。其在国际市场上的影响力也十分强烈。5. **国际影响力**：尽管票房表现优异，但《哪吒之魔童闹海》在国际市场的表现仍然较为突出，成为全球票房排名中的重要一员。6. **国际市场**：电影在国际市场上的知名度和影响力持续增长，尤其是在国际电影节上，许多国际电影节还在进行。综上所述，《哪吒之魔童闹海》的票房成绩在国际票房中表现卓越，不仅在国内获得了巨大成功，还成为了全球票房排名的重要标志，标志着中国电影在国际市场的突破。 [B]: 20字的摘要：20字的巨大成就是中国电影在国际市场取得的首次突破，标志着中国电影在国际市场的突破。 [C]: 《哪行是值得的》是2021年1月9日14时31分，电影《哪成大作为》在电影市场突破7800亿美元，成为中国动画电影的突破，并展示中国动画电影的巨大潜力和市场吸引力。 [D]: 我是一个参数很少的AI模型🥺，知识库较少，无法直接回答您的问题，换个问题试试吧👋 ``` 🙋‍Directly feeding all the above questions and model responses to GPT-5.4 Thinking for review and ranking: Detailed Review ``` /* Detailed source-code truncated for AI context efficiency. */ ``` | Rank | Model | Accuracy (30 pts) | Completeness (30 pts) | Logic (20 pts) | Code Quality (20 pts) | Total (100 pts) | |------|-------|--------------------|-----------------------|----------------|----------------------|-----------------| | 1 | B | 11 | 23 | 16 | 18 | 68 | | 2 | D | 25 | 19 | 15 | 3 | 62 | | 3 | A | 10 | 21 | 13 | 17 | 61 | | 4 | C | 8 | 6 | 5 | 2 | 21 | ### 👉 Comprehensive Evaluation 2 Subjectively, I would rank `minimind-3-moe` first, `chatlm-mini-chinese` second, `minimind-3` third, and `baby-llama2-chinese` fourth. Although `B` has severe hallucinations in knowledge accuracy (such as giant pandas eating seafood), it stands out for its fluent expression, clear structure, and correct code implementation, giving it the highest overall output quality. `D` clearly leads in factual knowledge (Newton in 1687, the source of the Yangtze River, etc. are all correct), but its code answer reverses the condition and becomes completely unusable, and it directly gives up on the summarization task, which significantly lowers its score. `A` is close to `B` in style and its code is also usable, but both its stability and factual accuracy are worse than `B`; it is a typical case of "being able to say something about everything, but fabricating details upon closer inspection". `C` has obvious gaps in factuality, elaboration ability, and overall readability, only occasionally answering the simplest factual questions correctly. It is worth noting that `D` and `A` have very close total scores (62 vs 61), but their strengths and weaknesses are almost complementary: `D` wins on knowledge accuracy (25 vs 10), while `A` wins on coding ability (17 vs 3). This also reflects a typical phenomenon of small-parameter models — under a limited parameter budget, "writing well" and "writing correctly" are often hard to achieve simultaneously. --- ## Ⅳ RoPE Length Extrapolation MiniMind supports length extrapolation of RoPE positional encoding through the YaRN algorithm, enabling the model to more stably handle text sequences that exceed the training length. When using the native torch model for inference with `eval_llm.py`, simply add the `--inference_rope_scaling` parameter to enable RoPE extrapolation: ```bash python eval_llm.py --weight full_sft --inference_rope_scaling ``` For models in `Transformers` format, the following configuration can be added to `config.json` to achieve length extrapolation: ```json "rope_scaling": { "type": "yarn", "factor": 16.0, "original_max_position_embeddings": 2048, "beta_fast": 32.0, "beta_slow": 1.0, "attention_factor": 1.0 } ``` Below, using MiniMind as an example, we use vernacular text from *Journey to the West* of different lengths as input, comparing the perplexity (PPL) changes before and after enabling RoPE scaling. It can be seen that in long-text scenarios, the model's PPL significantly decreases after enabling YaRN extrapolation: > PPL comparison of MiniMind before and after enabling YaRN at different text lengths --- ## Ⅴ Objective Evaluation This section reports benchmark results on several micro-scale language models. The selected benchmarks are C-Eval, CMMLU, ARC-Easy, PIQA, OpenBookQA, HellaSwag, and Social-IQa; all except the first two are English benchmarks. The evaluation framework chosen is [lm-evaluation](https://github.com/EleutherAI/lm-evaluation-harness) ```bash # Installation git clone https://github.com/EleutherAI/lm-evaluation-harness cd lm-evaluation-harness && pip install -e . ``` ```bash # Start testing # Datasets used: ceval-valid/cmmlu/arc_easy/piqa/openbookqa/hellaswag/social_iqa # View supported datasets: lm_eval ls tasks # For instruction-tuned models, add --apply_chat_template during evaluation; for base models such as gpt2, you do not need it. HF_ENDPOINT=https://hf-mirror.com lm_eval --model hf --model_args pretrained="/path/to/model",dtype=auto --tasks "task" --batch_size 16 --device cpu --trust_remote_code --apply_chat_template ``` > Note: These multiple-choice benchmarks are usually not evaluated by asking the model to freely generate the full answer. Instead, given a context `y` and a set of candidate options `x`, the standard practice is to compare the conditional probability `p(x | y)` of each option and select the one with the highest score. If an option maps to a single token, comparing that token probability is enough; if it spans multiple tokens, a more standard approach is to compare the sum of conditional log-probabilities over the whole option. The candidates are not necessarily `A`, `B`, `C`, `D`; some datasets have only two options. In that sense, random guessing is already a fairly strong lower bound, and models at this scale do tend to stay close to it for quite a while. MiniMind is trained on far less data than the other models listed here, and its training mix is heavily skewed toward Chinese, so its English performance is relatively weak. It is also not specifically aligned to this multiple-choice evaluation format by default, so its performance is relatively weak, and the results are only for fun: | model name | from | params | zh (ceval / cmmlu) | en (arc / piqa / obqa / hellaswag / siqa) | |---|---|---|---|---| | minimind-3 | current | 64M | 24.89 / 25.38 | 28.49 / 50.65 / 23.60 / 28.28 / 34.19 | | minimind-3-moe | current | 198M | 25.48 / 24.32 | 27.74 / 50.71 / 26.20 / 27.43 / 34.03 | | minimind-3-exam | current | 64M | 30.98 / 26.12 | 35.61 / 56.26 / 24.20 / 28.40 / 34.19 | | [Steel-LLM](https://huggingface.co/gqszhanshijin/Steel-LLM) | ZhanShiJin | 1121M | 24.89 / 25.32 | 39.69 / 65.13 / 26.00 / 35.73 / 39.15 | | [gpt2-medium](https://huggingface.co/openai-community/gpt2-medium) | OpenAI | 360M | 23.18 / 25.00 | 43.60 / 66.38 / 30.20 / 39.38 / 39.10 | | [TinyLlama-1.1B](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0) | TinyLlama | 1100M | 25.71 / 25.03 | 54.80 / 74.43 / 35.60 / 60.38 / 43.09 | | [SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct) | HuggingFace | 135M | 24.44 / 24.71 | 58.50 / 68.17 / 32.80 / 43.15 / 39.46 | | [Aquila-135M](https://huggingface.co/BAAI/Aquila-135M-Instruct) | BAAI | 135M | 25.19 / 25.10 | 54.59 / 67.52 / 34.40 / 41.67 / 39.66 | **Additional note (source / no contamination / reproduction)** minimind-3-exam is not a larger base model, and it contains little if any new knowledge. It is simply minimind-3 after a lightweight LoRA alignment on [lora_exam.jsonl](https://huggingface.co/datasets/jingyaogong/minimind_dataset/blob/main/lora_exam.jsonl), with [lora_exam_768.pth](https://huggingface.co/jingyaogong/minimind-3-pytorch/resolve/main/lora_exam_768.pth) merged back into the base model. This alignment data is sampled from the test subsets of ceval and English mmlu, with additional prefix/suffix augmentation. Its purpose is to align the context and option format commonly seen in multiple-choice evaluation, rather than to teach the answers. The 7 benchmarks used in this section have no sample overlap with the alignment data above, so this result can be regarded as free of data contamination. By contrast, if one fine-tunes directly on overlapping data, the scores of a small model can become heavily distorted; for example, minimind-3 once reached about 97% accuracy on contaminated ceval / cmmlu subsets, but such numbers are not meaningful. What this experiment suggests is simple: for this kind of benchmark, the bottleneck of a small model may not lie entirely in knowledge itself, but also in whether the input format is aligned. With only a small amount of format alignment, minimind-3-exam improves by about 2.9 percentage points on average across the 7 tasks above. --- # 📌 Others ## 🔧 Model Conversion * [./scripts/convert_model.py](./scripts/convert_model.py) can be used for conversion between `torch` and `transformers` model formats. * Unless otherwise specified, models released from the mainline `MiniMind` series are usually provided in `Transformers` format. If you use native `torch` weights, please run `torch2transformers` conversion first. ## 🖥️ API Service Interface Based on MiniMind * [./scripts/serve_openai_api.py](./scripts/serve_openai_api.py) provides a lightweight chat service compatible with the OpenAI API, making it easy to connect your own models to third-party UIs such as FastGPT, OpenWebUI, Dify, etc. * The API server additionally supports fields such as `reasoning_content`, `tool_calls`, and `open_thinking`, making it suitable for Tool Calling / Thinking scenarios. * After downloading model weights from [HuggingFace](https://huggingface.co/collections/jingyaogong/minimind-66caf8d999f5c7fa64f399e5), the directory structure example is as follows: ``` minimind (root dir) ├─ (e.g., minimind-3) | ├── config.json | ├── generation_config.json | ├── model_minimind.py (optional, depending on the export format) | ├── pytorch_model.bin or model.safetensors | ├── special_tokens_map.json | ├── tokenizer_config.json | ├── tokenizer.json ``` * Start the server ```bash cd scripts && python serve_openai_api.py ``` * Test the service interface ```bash cd scripts && python chat_api.py ``` * API request example (compatible with OpenAI API format) ```bash curl http://localhost:8998/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "model-identifier", "messages": [ { "role": "user", "content": "世界上最高的山是什么？" } ], "temperature": 0.7, "max_tokens": 1024, "stream": true, "open_thinking": true }' ``` ## [SGLang](https://github.com/sgl-project/sglang) SGLang is a high-performance LLM inference engine with optimizations such as RadixAttention and continuous batching, providing lower latency and higher throughput. > ⚠️ Requires a CUDA environment, use as needed. You can also select SGLang as the rollout / inference engine in RL training scripts to improve training throughput. Launch the model as an OpenAI-compatible API server: ```bash python -m sglang.launch_server --model-path /path/to/model --attention-backend triton --host 0.0.0.0 --port 8998 ``` ## [vllm](https://github.com/vllm-project/vllm) vLLM is a widely used efficient inference framework for rapid LLM deployment, with a good balance between memory efficiency and throughput. > ⚠️ Requires a CUDA environment, use as needed. Launch the model as an OpenAI-compatible API server: ```bash vllm serve /path/to/model --model-impl transformers --served-model-name "minimind" --port 8998 ``` ## [llama.cpp](https://github.com/ggerganov/llama.cpp) llama.cpp is a lightweight and practical C++ inference framework that can be used directly from the command line. It supports multi-threaded inference and several GPU acceleration options. **Directory structure**: It is recommended to place `llama.cpp` and the model directory at the same level path ``` parent/ ├── project/ # your project directory │ ├── minimind-model/ # HuggingFace-format model directory │ │ ├── config.json │ │ ├── model.safetensors │ │ └── ... │ └── ... └── llama.cpp/ # llama.cpp directory ├── build/ ├── convert_hf_to_gguf.py └── ... ``` 0. Refer to the `llama.cpp` official documentation to complete installation (dependencies such as `cmake`, etc.) 1. Insert at the end of the `get_vocab_base_pre` function in `convert_hf_to_gguf.py`: ```python # Add MiniMind tokenizer support. A compatible fallback such as qwen2 can be reused temporarily. if res is None: res = "qwen2" ``` 2. Convert the HuggingFace format minimind model to GGUF: ```bash # Run this under the llama.cpp directory. The GGUF file will be generated in the model directory. python convert_hf_to_gguf.py /path/to/minimind-model ``` 3. Quantize the model (optional) ```bash ./build/bin/llama-quantize /path/to/model/xxxx.gguf /path/to/model/xxxx.q8.gguf Q8_0 ``` 4. Command-line inference test ```bash ./build/bin/llama-cli -m /path/to/model/xxxx.gguf ``` ## [ollama](https://ollama.ai) Ollama is a commonly used tool for running large models locally. It supports many open-source LLMs and provides a simple workflow with low setup overhead. 1. Load a custom GGUF model via Ollama Create a new `minimind.modelfile` file in the model directory and write the following configuration template. You may replace the system prompt as needed: minimind.modelfile (template) ```text FROM /path/to/model/xxxx.gguf SYSTEM "你的名字叫MiniMind，你是一个乐于助人、知识渊博的AI助手。请用完整且友好的方式回答用户问题，当被问到名字时请回答MiniMind。" TEMPLATE """{{- if .Tools }}<|im_start|>system {{ if .System }}{{ .System }} {{ end }}# Tools You may call one or more functions to assist with the user query. You are provided with function signatures within XML tags: {{- range .Tools }} {"type": "function", "function": {{ .Function }}} {{- end }} For each function call, return a json object with function name and arguments within XML tags: {"name": , "arguments": } <|im_end|> {{ else if .System }}<|im_start|>system {{ .System }}<|im_end|> {{ end }} {{- range $i, $_ := .Messages }} {{- $last := eq (len (slice $.Messages $i)) 1 -}} {{- if eq .Role "user" }}<|im_start|>user {{ .Content }}<|im_end|> {{ else if eq .Role "assistant" }}<|im_start|>assistant {{ .Thinking }} {{ .Content }} {{- if .ToolCalls }} {{- range .ToolCalls }} {"name": "{{ .Function.Name }}", "arguments": {{ .Function.Arguments }}} {{- end }} {{- end }} {{- if not $last }}<|im_end|> {{ end }} {{- else if eq .Role "tool" }}<|im_start|>user {{ .Content }} <|im_end|> {{ end }} {{- if and (ne .Role "assistant") $last }}<|im_start|>assistant {{ if and $.IsThinkSet $.Think -}} {{ else -}} {{ end -}} {{ end }} {{- end }}""" PARAMETER repeat_penalty 1 PARAMETER stop "<|im_start|>" PARAMETER stop "<|im_end|>" PARAMETER temperature 0.9 PARAMETER top_p 0.9 PARAMETER num_ctx 8192 ``` 2. Load and name the local model ```bash ollama create -f minimind.modelfile minimind-local ``` 3. Start inference ```bash ollama run minimind-local ``` 📤 Push your model to Ollama Hub ```bash # 1. Rename the local model to your ollama-account/minimind tag ollama cp minimind-local:latest your_username/minimind:latest # 2. Push the model ollama push your_username/minimind:latest ``` ⭐️ You can also directly use the Ollama model I provide for a quick start: ```bash ollama run jingyaogong/minimind-3 >>> 你叫什么名字 我是一个语言模型... ``` ## [MNN](https://github.com/alibaba/MNN) MNN is an AI inference engine designed for edge devices, supporting lightweight deployment and high-performance inference for various open-source LLMs. 1. Model conversion ```bash cd MNN/transformers/llm/export # Export a 4-bit HQQ-quantized MNN model python llmexport.py --path /path/to/model --export mnn --hqq --dst_path /path/to/model-mnn ``` 2. Test on Mac or mobile devices ```bash ./llm_demo /path/to/model-mnn/config.json prompt.txt ``` Alternatively, test it with the mobile app. > For more details on these third-party frameworks, please refer to their official documentation. ## 👨‍💻 More Content * [🔗Fine-tuning Diffusion Language Models from MiniMind-LLM](https://github.com/jingyaogong/minimind/discussions/618) * [🔗Description of the Model's generate Method](https://github.com/jingyaogong/minimind/discussions/611) * [🔗Training Linear Attention Models from MiniMind](https://github.com/jingyaogong/minimind/discussions/704) # 📌 Acknowledgments > [!NOTE] > If the `MiniMind` series of projects has been helpful to you, feel free to star ⭐ on GitHub > The documentation is long, and omissions are inevitable. Feedback via Issues or PRs is welcome to help improve the project together. > Your support and suggestions are important driving forces for the continuous iteration of this project! ## 🤝[Contributors](https://github.com/jingyaogong/minimind/graphs/contributors) ## 😊Acknowledgments Thanks to the following contributors for sharing training notes, data processing experience, tutorials, and code walkthroughs: * [@ipfgao](https://github.com/ipfgao): [🔗Training Step Records](https://github.com/jingyaogong/minimind/issues/26) * [@WangRongsheng](https://github.com/WangRongsheng): [🔗Large Dataset Preprocessing](https://github.com/jingyaogong/minimind/issues/39) * [@pengqianhan](https://github.com/pengqianhan): [🔗A Concise Tutorial](https://github.com/jingyaogong/minimind/issues/73) * [@RyanSunn](https://github.com/RyanSunn): [🔗Inference Process Learning Notes](https://github.com/jingyaogong/minimind/issues/75) * [@Nijikadesu](https://github.com/Nijikadesu): [🔗Breaking Down Project Code in Interactive Notebook Format](https://github.com/jingyaogong/minimind/issues/213) Thanks also to the following papers and projects: - [https://github.com/meta-llama/llama3](https://github.com/meta-llama/llama3) - [https://github.com/karpathy/llama2.c](https://github.com/karpathy/llama2.c) - [https://github.com/DLLXW/baby-llama2-chinese](https://github.com/DLLXW/baby-llama2-chinese) - [DeepSeek-V2](https://arxiv.org/abs/2405.04434) - [https://github.com/charent/ChatLM-mini-Chinese](https://github.com/charent/ChatLM-mini-Chinese) - [https://github.com/wdndev/tiny-llm-zh](https://github.com/wdndev/tiny-llm-zh) - [Mistral-MoE](https://arxiv.org/pdf/2401.04088) - [https://github.com/Tongjilibo/build_MiniLLM_from_scratch](https://github.com/Tongjilibo/build_MiniLLM_from_scratch) - [https://github.com/jzhang38/TinyLlama](https://github.com/jzhang38/TinyLlama) - [https://github.com/AI-Study-Han/Zero-Chatgpt](https://github.com/AI-Study-Han/Zero-Chatgpt) - [https://github.com/xusenlinzy/api-for-open-llm](https://github.com/xusenlinzy/api-for-open-llm) - [https://github.com/HqWu-HITCS/Awesome-Chinese-LLM](https://github.com/HqWu-HITCS/Awesome-Chinese-LLM) ## 🫶Supporters ## 🎉 MiniMind Related Achievements This model has served as a stepping stone for several gratifying research outcomes. Thanks to the researchers for their recognition: - ECG-Expert-QA: A Benchmark for Evaluating Medical Large Language Models in Heart Disease Diagnosis [[arxiv](https://arxiv.org/pdf/2502.17475)] - Binary-Integer-Programming Based Algorithm for Expert Load Balancing in Mixture-of-Experts Models [[arxiv](https://arxiv.org/pdf/2502.15451)] - LegalEval-Q: A New Benchmark for The Quality Evaluation of LLM-Generated Legal Text [[arxiv](https://arxiv.org/pdf/2505.24826)] - On the Generalization Ability of Next-Token-Prediction Pretraining [[ICML 2025](https://openreview.net/forum?id=hLGJ1qZPdu)] - 《从零开始写大模型：从神经网络到Transformer》王双、牟晨、王昊怡 编著 - 清华大学出版社 - FedBRB: A Solution to the Small-to-Large Scenario in Device-Heterogeneity Federated Learning [[TMC 2025](https://ieeexplore.ieee.org/abstract/document/11168259)] - SKETCH: Semantic Key-Point Conditioning for Long-Horizon Vessel Trajectory Prediction [[arxiv](https://arxiv.org/pdf/2601.18537)] - A Built-in Crypto Expert for Artificial Intelligence: How Far is the Horizon? [[IACR ePrint 2026](https://eprint.iacr.org/2026/411.pdf)] - RetryTrigger: Intelligent Inference Duplication for Enhancing LLM Resilience to Hardware Transient Faults [[FITEE 2026](https://ieeexplore.ieee.org/abstract/document/11479682)] - In progress... # 🎓 Citation If `MiniMind` has been helpful to your research or work, feel free to cite: ```bibtex @misc{minimind, title = {MiniMind: Train a Tiny LLM from Scratch}, author = {Jingyao Gong}, year = {2024}, url = {https://github.com/jingyaogong/minimind}, note = {GitHub repository} } ``` # ⚖️ License This project is open-sourced under the [Apache License 2.0](LICENSE).