Hao AI Lab fixes SGLang multi-GPU communication misalignment with vstack
Hao AI Lab used vstack to locate and fix a multi-GPU rank misalignment in SGLang caused by the prefill delayer and interval min batch size under DP attention: busy ranks skipped prefill and went straight into the per-round synchronization, while idle ranks entered the delayer negotiation, and the communication library then aborted. After making ranks with delayed prefill still participate in the negotiation with zero prefill, all 30 cross-threshold runs on GB200 completed with no aborts.
Body
(5/8) Multi-GPU rank mismatch: under DP attention with the prefill delayer and interval min batch size, two DP ranks enter different collectives and exit.
This breaks multi-GPU lockstep: every rank sharing a communication group must run the exact same collectives in the exact same order each step.
Each rank checks its own running batch size when deciding whether to defer prefill. When one rank meets the threshold and another sits below it, the busy rank skips prefill and goes straight to the per-round sync while the idle rank enters the prefill-delayer negotiation on the same CPU group, causing the communication library to abort.
Having a deferring rank still join the delayer negotiation with zero prefill keeps every collective aligned, completing 30 of 30 cross-threshold runs on GB200 with no abort.
PR: https://t.co/lhGKES67Cw
AI Trends