llama.cpp 通过 ggml RPC 后端在异构设备上分发推理
llama.cpp can distribute inference on heterogeneous devices through the ggml RPC backend
llama.cpp 作者 Georgi Gerganov 表示，llama.cpp 可以通过 ggml RPC 后端在异构设备上分发推理，目前属于高级设置，他预计未来会降低普通用户的使用门槛。被引用的演示显示，MiMo 2.6 Flash 的原生 mxfp4 权重可跨 RTX 6000 GPU 与 M5 笔记本运行，在 10 GbE 下达到 40 tokens/sec，llama.cpp 开箱支持。
查看原帖It's crazy that I can run MiMo 2.6 Flash across my RTX 6000 GPU and my M5 laptop at 40 tokens/sec over 10 GbE 🤯 These are the native mxfp4 weights of a state-of-the-art model, on heterogeneous hardware. Supported out of the box in llama.cpp. https://t.co/5ElpeK1Wlt
正文
当前显示原文，中文全文译文尚未就绪。
llama.cpp can distribute inference on heterogeneous devices through the ggml RPC backend
It's an advanced setting but I think with time we'll make it more accessible to regular users.