Prime Intellect 开源 OpenDiLoCo，实现跨洲分布式低通信训练
Prime Intellect 开源 OpenDiLoCo，这是 DeepMind DiLoCo 低通信分布式训练方法的可复现实现，代码基于 Hivemind 与 PyTorch FSDP，已发布在 GitHub。团队在加拿大、芬兰和美国两地共四个 DiLoCo worker（各 8 张 H100）上训练 1.1B 参数模型，跨两大洲三国，计算利用率保持 90-95%，all-reduce 仅占训练时间 6.9%。相比原论文 400M 参数规模，该方法被扩展到 1.1B 参数，并验证伪梯度可用 FP16 all-reduce 而不损失性能。
efficiency. We also demonstrated that DiLoCo pseudo gradients can be effectively all-reduced in FP16 without any performance degradation. For more details, check out the paper on arXiv. The original DiLoCo paper by DeepMind only experimented with model sizes of up to 400 million parameters. In our work, we scale the method to a model with 1.1 billion parameters. We adopt the same hyperparameters as TinyLlama and use a total batch size containing 8 million tokens (batch size of 8192 with sequenc
匹配字段正文