NVIDIA Megatron Core 如何将比特级确定性预训练开销降至约 2%
NVIDIA 在 Megatron Core 中通过记录每 rank 有序张量指纹来定位并修复大规模预训练中的非确定性,在三个 Nemotron 工作负载上把确定性开销降至约 2%(2,432 块 GPU),并在 800 步内保持比特级确定性。相关改动包括让 grouped-GEMM 写入方使用私有输出槽并采用固定归约顺序,同时通过 recipe 校验、kernel 测试和模块校验覆盖 FP8、FP4 等并行配置以防回归。
ate across parallelism configurations, including FP8 and FP4. Independent end-to-end runs then verify that full-precision training metrics remain bitwise identical For current coverage and known gaps, see determinism status, operation catalog, and kernel testing guide. Common sources of nondeterminism Use Table 2 to identify likely causes of nondeterminism and choose the next diagnostic step. Bitwise determinism makes large-scale pretraining easier to debug and resume. The Nemotron case study sh
匹配字段正文







