Hao AI Lab uses vstack to discover SGLang streaming stop-string truncation defect
Hao AI Lab uses vstack to discover a streaming stop-string truncation defect in SGLang: when a byte-level tokenizer splits a multi-byte character across multiple tokens, a trailing decoded U+FFFD breaks the prefix check, causing the beginning of the stop string to be sent to the client prematurely. Tested on GB200 with Qwen3-4B and default parameters, 8 out of 8 streaming comparisons across the chat and completions endpoints were not truncated, and after additionally checking the text with the trailing U+FFFD removed, this dropped to 0. This issue is one of 20+ issues raised in collaboration with the SGLang community.
Body
(4/8) Streaming stop-string trimming issue: in streaming mode, SGLang can send the client the start of a stop string when a later character is split across tokens.
This breaks stop-string trimming: streaming and non-streaming generation must stop before the exact same stop string.
Byte-level tokenizers often split emoji and CJK characters across multiple tokens. SGLang holds back streaming text while the decoded tail matches a stop-string prefix, but a token carrying only the first bytes of a multi-byte character decodes to U+FFFD, breaking that prefix check and flushing the printable head of the stop string early.
On GB200 with Qwen3-4B under default flags, the stop-string prefix was emitted untrimmed in 8 of 8 streaming comparisons across chat and completions endpoints, and 0 of 8 after also checking the tail stripped of trailing U+FFFD.
PR: https://t.co/KpPlSIEjGM
AI Trends