Hao AI Lab used vstack to find 17 bugs in vLLM
Hao AI Lab used vstack to perform agent-based formal verification on vLLM, and has found 17 bugs (still increasing), publicly disclosing 5 of them along with fix PRs. When fine-grained prefix matching is enabled for hybrid models, vLLM marks cache blocks as reusable before they are written; the second request in the same step then copies all-zero blocks and continues generating. On GB200, all-zero blocks were reproduced 32/32 times before the fix and 0 times after the fix. Other issues include: under the queue limit, just 2 failed requests can cause the service to continuously return busy; streaming sessions write out of bounds to shared cache blocks; after in-place LoRA reload, new requests still use the old weights; and cross-instance image cache key collisions.
Body
(2/7) Silent wrong output: hybrid models can continue from an all-zero cache block, and then cache it.
This breaks prefix caching: a cache hit must give a request the same result that it would have computed itself.
On hybrid models with fine-grained prefix matching turned on, vLLM marks a cache block as reusable as soon as it plans the block, before the block is actually written. So when a second request in the same step reuses that block, it copies it too early and gets all zeros. The second request then continues from those zeros. Since the bad block stays in the cache, later requests can pick it up as well, while every response still looks normal.
On GB200, the copy source was all zeros in 32 of 32 trials before the fix, and 0 after. A small fix defers a request with an unwritten copy source to the next scheduler step, after the source is populated.
PR: https://t.co/NMPdqdCAxv