ModelBest MiniCPM-V 4.7 inference speed on iPhone increased from 3.38 to 20.51 tokens/s
ModelBest OpenBMB reposted a breakdown of inference optimization for MiniCPM-V 4.7 35B-A3B on the iPhone 18 Pro Max (12GB RAM), with decoding speed increasing from 3.38 tokens/s to 20.51 tokens/s, about 6.07 times the unoptimized baseline. The optimization path was, in order, expert prefetching to 6.57, scheduling and loading optimization to 11.34, mixed-precision quantization to 17.16, and then adding DSpark speculative decoding to reach 20.51 tokens/s; the model has 35B total parameters and about 3B active parameters, supports image, multi-image, and video understanding, and keeps most weights in flash storage for on-demand loading. The original post notes that these are still OpenBMB's own measurements.
View original postThis new-ish 35B multimodal model went from 3.38 → 20.51 tps on an iPhone. MiniCPM-V 4.7 35B-A3B was updated to decode faster on the iPhone by improving how it moves the model between storage and memory. Constraints 🧠 35B total / ~3B active 📱 iPhone 18 Pro Max — 12GB RAM 👁️ image + multi-image + video understanding Most of those 35B weights don't sit in RAM. The experts can stay in the phone's flash storage and get loaded when they're actually needed. Normally that's slow 🐌 SSD expert loading → 3.38 t/s So OpenBMB built Lookahead Expert Scheduling (LES) It predicts which experts the next layer will probably need and starts loading them while the current layer is still computing. Then they stacked the optimizations: 💾 baseline SSD loading → 3.38 tps 👀 expert lookahead → 6.57 ⚙️ optimized scheduling/loading → 11.34 🗜️ mixed-precision quantization → 17.16 🚀 + DSpark speculative decoding → 20.51 tps That's a 6.07× speedup over the same-phone unoptimized baseline. ⚠️ And these are still OpenBMB's own measurements. Link in ALT
View original
This new-ish 35B multimodal model went from 3.38 → 20.51 tps on an iPhone. MiniCPM-V 4.7 35B-A3B was updated to decode faster on the iPhone by improving how it moves the model between storage and memory. Constraints 🧠 35B total / ~3B active 📱 iPhone 18 Pro Max — 12GB RAM 👁️ image + multi-image + video understanding Most of those 35B weights don't sit in RAM. The experts can stay in the phone's flash storage and get loaded when they're actually needed. Normally that's slow 🐌 SSD expert loading → 3.38 t/s So OpenBMB built Lookahead Expert Scheduling (LES) It predicts which experts the next layer will probably need and starts loading them while the current layer is still computing. Then they stacked the optimizations: 💾 baseline SSD loading → 3.38 tps 👀 expert lookahead → 6.57 ⚙️ optimized scheduling/loading → 11.34 🗜️ mixed-precision quantization → 17.16 🚀 + DSpark speculative decoding → 20.51 tps That's a 6.07× speedup over the same-phone unoptimized baseline. ⚠️ And these are still OpenBMB's own measurements. Link in ALT
Body
Love this breakdown! 🔥 From 3.38 to 20.51 tokens/s on an iPhone — a great example of how inference optimizations can unlock powerful on-device AI.
Thanks for highlighting the engineering behind MiniCPM-V 4.7! 🚀💙