Hao AI Lab uses vstack to find out-of-bounds writes to shared cache blocks from streaming sessions in vLLM
In the fourth entry of the vstack agent-based formal verification series, Hao AI Lab points out that when vLLM processes streaming input, it only checks the length of each new chunk and not the length of the entire session, so a session may exceed the model's maximum length. vLLM does not reserve memory for these extra tokens, and their data overflows beyond the session's memory and may land in shared cache blocks; subsequent requests that reuse such a block will get different outputs without any error being reported. Limiting worker writes and discarding token data that exceeds the limit can prevent this issue, and the relevant PR has been submitted.
Why read this
AI score 51
This content discloses the issue of out-of-bounds writes to shared cache blocks from streaming input in vLLM, allowing readers to understand how long sessions exceeding the model's length limit can break memory isolation.
Topics
Body
(4/7) Silent out-of-bounds write: a long streaming session overwrites a shared cache block.
This breaks memory isolation: a request must never write outside its own memory.
With streaming input, vLLM checks each new chunk's length but not the whole session's, so a session can exceed the model's maximum length. vLLM reserves no memory for those extra tokens, so their data spills outside the session's memory and can land in a shared cache block. A later request reusing that block can get different output, with no error.
A long streaming session past the model's length limit overwrote a shared prefix cache block. Bounding the worker write prevents that, dropping token data past the limit.
PR: https://t.co/vjh9CHLCqh