AI Trends

Hao AI Lab uses vstack to discover SGLang request cancellation loss issue

UCSD Hao AI LabUpdated Nov 18 02:32

AI summary

Hao AI Lab used vstack to discover a cancellation loss issue under DP attention in SGLang: a fast abort may arrive before its generation request and be dropped, causing the request to continue running until max_new_tokens. On GB200, 33 of 204 aborts issued within 20 ms were lost and ran to max_new_tokens; after changing to transmitting aborts in order on each rank's work socket, this dropped to 0/204. A fix PR has been submitted for this issue.

Body

(6/8) Lost cancellation: under DP attention, a rapid cancel can overtake its own generation request and be dropped, leaving the request running to max_new_tokens.

This breaks cancellation finality: once an abort request succeeds, the cancelled request must not run to completion.

With the default DP control routing, a generation request goes directly to its owning rank while an abort goes to rank 0 and is broadcast across ranks, with no ordering guarantee between the two paths. If the abort arrives first, the scheduler has not seen the request yet, drops the abort, and runs the request to its token limit on the next poll.

On GB200, 33 of 204 aborts sent within 20 ms were lost and ran to max_new_tokens, dropping to 0 of 204 once aborts travel in order on each rank's work socket.

PR: https://t.co/wI4nH8vUq7

Source

UCSD Hao AI Lab