AI 热点

Hao AI Lab 用 vstack 发现 SGLang 流式 stop-string 截断缺陷

UCSD Hao AI Lab更新于 11月18日 02:29

(4/8) Streaming stop-string trimming issue: in streaming mode, SGLang can send the client the start of a stop string when a later character is split across toke

AI 导读

Hao AI Lab 用 vstack 在 SGLang 中发现流式 stop-string 截断缺陷:字节级 tokenizer 把多字节字符拆到多个 token 时,尾部解码出 U+FFFD 会破坏前缀检查,导致 stop string 开头被提前发给客户端。在 GB200 上用 Qwen3-4B、默认参数测试,chat 与 completions 端点 8 次流式对比中 8 次未截断,额外检查去掉尾部 U+FFFD 的文本后降为 0 次。该问题属其与 SGLang 社区合作提出的 20+ 个问题之一。

正文

(4/8) Streaming stop-string trimming issue: in streaming mode, SGLang can send the client the start of a stop string when a later character is split across tokens.

This breaks stop-string trimming: streaming and non-streaming generation must stop before the exact same stop string.

Byte-level tokenizers often split emoji and CJK characters across multiple tokens. SGLang holds back streaming text while the decoded tail matches a stop-string prefix, but a token carrying only the first bytes of a multi-byte character decodes to U+FFFD, breaking that prefix check and flushing the printable head of the stop string early.

On GB200 with Qwen3-4B under default flags, the stop-string prefix was emitted untrimmed in 8 of 8 streaming comparisons across chat and completions endpoints, and 0 of 8 after also checking the tail stripped of trailing U+FFFD.

PR: https://t.co/KpPlSIEjGM

来源

UCSD Hao AI Lab