Understand this story
The Google Developers Blog introduces that Google has implemented the Zero-TIM zero training-inference mismatch scheme in Tunix, achieving bit-for-bit identical logprobs for the same token between the sampler and the trainer on 64 TPU v5p chips (DP8×TP8), eliminating the numerical discrepancy between training and inference. The specific approach includes fixing the accumulation order within shards, explicitly specifying the reduction order across shards, unifying the logprob computation path for sampling actions, and writing custom differentiation rules for the serving-side kernel. Validated on FrozenLake with Qwen3-8B over 200 steps, Zero-TIM's sampler and trainer logprobs show a mean absolute difference of zero, and the observed solve rate is higher than that of standard training and TIS.
AI-generated from public reports
Latest development · Report
Google uses Zero-TIM on TPUs to eliminate numerical discrepancies between training and inferenceDevelopments
1 confirmed developments
Google implements Zero-TIM training-inference numerical alignment in Tunix
Source:Google Developers Blog
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 16
Google Developers BlogFirst-party
Google uses Zero-TIM on TPUs to eliminate numerical discrepancies between training and inference
The Google Developers Blog introduces Zero-TIM, implemented in Tunix, which on 64 TPU v5p chips (DP8×TP8) makes the sampler and trainer produce bitwise-identical logprobs for the same token, eliminating numerical discrepancies between training and inference. The approach includes fixing the accumulation order within shards, explicitly specifying the reduction order across shards, unifying the logprob computation path for sampling actions, and writing custom differentiation rules for the server-side kernel. Validated on FrozenLake with Qwen3-8B over 200 steps, Zero-TIM's sampler and trainer logprobs have a mean absolute difference of zero, and the observed solve rate is higher than that of standard training and TIS.