AI Trends

Hao AI Lab used vstack to find 20+ concurrency bugs in SGLang

UCSD Hao AI LabUpdated Nov 18 02:32

AI summary

Hao AI Lab and the SGLang community used vstack to identify 20+ concurrency and multi-step state bugs in SGLang and submitted fix PRs. Among them, the weight update race condition, before the fix on GB200, caused in-flight steps to compute tokens, logprobs, and KV with the new weights in 8/8 trials; after the fix, this was 0/8. During the 364.69 seconds the decode server was stuck, /health still returned 200 in 363/363 cases. Fixes include adding a GPU stream wait before parameter writes and restricting only new preallocations, among others.

Body

(2/8) Weight-update race condition: after an in-place pause in an RL loop, a weight update can overwrite weights while an in-flight step is still reading them.

This breaks rollout policy isolation: a forward step launched before a pause must finish reading the old weights before a weight update changes them.

With the default overlap scheduler, an in-place pause returns right away while the already-launched step keeps running on the GPU's forward stream. Parameter writes run on the scheduler stream, whose barrier only waits for input metadata reads before the model layers, so new weights can land mid-step with no error.

On GB200, the in-flight step computed its token, logprob, and KV from the updated weights in 8 of 8 trials before the fix, and 0 of 8 after. Adding a GPU stream wait before parameter writes ensures the forward step finishes first.

PR: https://t.co/akyxGsdGu2

Source

UCSD Hao AI Lab