面壁智能 MiniCPM-V 4.7 在 iPhone 上实现 20.51 tokens/s 端侧推理
📱 A 35B MoE model running on a phone at over 20 tokens/s.
面壁智能 OpenBMB 公布 MiniCPM-V 4.7 的端侧推理方案，在 iPhone 18 Pro Max（12GB）上平均解码速度达 20.51 tokens/s，相比同一设备未优化的 SSD 专家加载基线提升 6.07 倍。该方案名为 Lookahead Expert Scheduling（LES），通过跨层路由预测预取下一层可能需要的专家、并行加载剩余专家，并对专家权重做连续存储重排和逐层混合精度量化，同时配合 DSpark 投机采样。模型为 35B 总参数、每 token 激活 3B 的 MoE 架构，原始模型架构与专家路由保持不变，无需额外训练。
查看原帖35B multimodal AI. Running locally on an iPhone. Now at over 20 tokens/sec. 🚀 MiniCPM-V 4.7 achieves 20.51 tokens/sec average decoding speed on iPhone 18 Pro Max (12GB) — a 6.07× speedup over the same device’s unoptimized SSD expert-loading baseline. ⚡ Behind the speed: Lookahead Expert Scheduling (LES), optimized expert loading, fine-grained…
展开完整原文
35B multimodal AI. Running locally on an iPhone. Now at over 20 tokens/sec. 🚀 MiniCPM-V 4.7 achieves 20.51 tokens/sec average decoding speed on iPhone 18 Pro Max (12GB) — a 6.07× speedup over the same device’s unoptimized SSD expert-loading baseline. ⚡ Behind the speed: Lookahead Expert Scheduling (LES), optimized expert loading, fine-grained quantization, and DSpark speculative decoding. 📱 Watch the new on-device demo below. Powerful multimodal AI, right in your pocket. Try it 👇 🔗 GitHub: https://t.co/msjm2QkTGp 🤗 Hugging Face: https://t.co/W95QKGh0wY LES: https://t.co/6Ul7qXBItU #MiniCPM #OpenBMB
正文
当前显示原文，中文全文译文尚未就绪。
📱 A 35B MoE model running on a phone at over 20 tokens/s.
Meet Lookahead Expert Scheduling (LES), the inference technology behind MiniCPM-V 4.7’s efficient on-device performance.
🧠 The challenge: 35B total parameters, 3B activated per token, and limited device memory. Keeping experts in flash storage makes the model fit, but loading their weights can leave computation waiting.
LES co-optimizes expert loading, compute scheduling, and weight access:
⚡ Expert prefetch and compute scheduling: Cross-layer routing prediction prefetches experts the next layer is likely to need. Experts selected by the actual router that are already loaded compute first, while the remaining required experts load in parallel. Prediction only guides prefetch and scheduling: every required expert is still loaded and used. No additional training is needed.
💾 Weight access optimization: Repack expert weights into contiguous storage to reduce scattered reads, and apply precise layer-wise mixed-precision quantization to reduce weight storage and transfer costs while preserving model capability.
We also adapted DSpark speculative sampling to work alongside LES in the on-device inference pipeline.
In our iPhone 18 Pro Max tests, the full system reaches 20.51 tokens/s, up from a 3.38 tokens/s on-demand loading baseline: 6.07× faster. 🔗 https://t.co/kFD3X4RN0P
The original model architecture and expert routing stay intact. Smarter scheduling makes larger models practical on-device.
#MiniCPM #OnDeviceAI #OpenSource