AI Trends

Hao AI Lab uses vstack to discover SGLang decode server hang of 364 seconds

UCSD Hao AI LabUpdated Nov 18 02:32

AI summary

Hao AI Lab used vstack to discover a PD-disaggregated decode server hang issue in SGLang: a single aborted request freezes the transfer queue, while /health still returns 200. On GB200, the decode server stalled for 364.69 seconds, and all 363 health checks passed; gating only new preallocations allows already-arrived transfers to complete and resolves the hang. This issue is one of 20+ issues and fix PRs proposed in collaboration with the SGLang community.

Body

(3/8) Decode server stall: on a PD-disaggregated decode server, a single retracted request freezes the transfer queue while health checks still pass.

This breaks liveness: requests whose KV cache has already arrived must be able to run and release their reserved memory so retracted work can resume.

Under KV pressure, the decode scheduler retracts a request and returns early every step to hold off new preallocations. That early return also skips polling the transfer queue, so arrived transfers keep holding their pages and decode reserves while the retracted request never fits, even as /health still returns 200.

On GB200, the decode server stalled for 364.69 s while /health answered 200 in 363 of 363 probes. Gating only new preallocations lets arrived transfers finish and clears the stall.

PR: https://t.co/55VVQdpLSY

Source

UCSD Hao AI Lab