AI News

Trending story

Hao AI Lab uses vstack to perform agent-based formal verification of vLLM

5 reports5 story updatesUpdated Nov 14 03:13

Understand this story

AI overview

Hao AI Lab, in collaboration with the vLLM project, used vstack to conduct agent-based formal verification of vLLM, and has found 17 bugs (and counting), ranging from silent incorrect output to server lockups. The team publicly disclosed the causes and fix PRs for 5 of these bugs; for example, a hybrid model can continue generating from an all-zero cache block, and when a failed request under the queue limit does not return its slot, the server rejects all new requests. The relevant fixes are being submitted upstream. The fourth item in the series points out that when vLLM handles streaming input, it only checks the length of each new chunk rather than the length of the entire session, so a session may exceed the model's maximum length, extra token data overflows and may fall into a shared cache block, and subsequent requests that reuse that block receive different output without any error being reported; limiting worker writes and discarding token data beyond the limit can prevent this problem, and the relevant PR has been submitted.

AI-generated from public reports

Latest development · Report

Hao AI Lab uses vstack to perform agentic formal verification on vLLM, finding more than 17 bugs

Developments

2 confirmed developments

  1. Hao AI Lab uses vstack to perform agentic formal verification on vLLM

    Source:UCSD Hao AI Lab
  2. Hao AI Lab uses vstack to perform agentic formal verification on vLLM

    Source:UCSD Hao AI Lab

Reports and discussion timeline

Browse reports and discussion by time. Their evidence levels remain distinct.

Nov 14

  1. UCSD Hao AI LabFirst-party

    Hao AI Lab uses vstack to perform agentic formal verification on vLLM, finding more than 17 bugs

    Hao AI Lab used vstack to perform agentic formal verification on vLLM and has found 17 bugs (and counting), publicly disclosing 5 of them along with the corresponding fix PRs. The issues include hybrid models continuing to generate from all-zero cache blocks, a small number of failed requests under the queue limit causing the server to continuously return busy, out-of-bounds writes to shared cache blocks in long streaming sessions, new requests still using old weights after a LoRA adapter is reloaded in place, and different images hitting the same cache key when two servers share an external KV cache. The relevant fixes are being submitted upstream, and the team says it will continue to work with the vLLM team and the open source community.

  2. UCSD Hao AI LabFirst-party

    Hao AI Lab uses vstack to find out-of-bounds writes to shared cache blocks from streaming sessions in vLLM

    In the fourth entry of the vstack agent-based formal verification series, Hao AI Lab points out that when vLLM processes streaming input, it only checks the length of each new chunk and not the length of the entire session, so a session may exceed the model's maximum length. vLLM does not reserve memory for these extra tokens, and their data overflows beyond the session's memory and may land in shared cache blocks; subsequent requests that reuse such a block will get different outputs without any error being reported. Limiting worker writes and discarding token data that exceeds the limit can prevent this issue, and the relevant PR has been submitted.

  3. UCSD Hao AI LabFirst-party

    New requests still run old weights after vLLM reloads LoRA in place

    Hao AI Lab used the vstack agent for formal verification and found that after vLLM reloads a LoRA adapter in place, new requests may still load the old weights, while vLLM reports the new version. The reason is that the reload keeps the adapter name and ID but points to the new file; requests already admitted before the reload still point to the old file, and when the number of adapters in vLLM's memory exceeds the limit and a reload is needed, the file is taken from that old request. The fix is to track the latest successful reload of each adapter and ensure loading from the new file.

  4. UCSD Hao AI LabFirst-partySelected

    Hao AI Lab uses vstack to perform agentic formal verification on vLLM, finding 17 bugs

    Hao AI Lab, in collaboration with the vLLM project, used vstack to perform agentic formal verification on vLLM and has found 17 bugs (and counting), ranging from silent incorrect outputs to server lockups. The team disclosed the root causes and fix PRs for 5 of these bugs, for example, hybrid models can continue generating from all-zero cache blocks, and failed requests under the queue limit do not return their slots, causing the server to reject all new requests. The relevant fixes are being submitted upstream.

  5. UCSD Hao AI LabFirst-party

    Hao AI Lab used vstack to find 17 bugs in vLLM

    Hao AI Lab used vstack to perform agent-based formal verification on vLLM, and has found 17 bugs (still increasing), publicly disclosing 5 of them along with fix PRs. When fine-grained prefix matching is enabled for hybrid models, vLLM marks cache blocks as reusable before they are written; the second request in the same step then copies all-zero blocks and continues generating. On GB200, all-zero blocks were reproduced 32/32 times before the fix and 0 times after the fix. Other issues include: under the queue limit, just 2 failed requests can cause the service to continuously return busy; streaming sessions write out of bounds to shared cache blocks; after in-place LoRA reload, new requests still use the old weights; and cross-instance image cache key collisions.

Related stories

  1. Related storiesHao AI Lab launches vstack and veriserve
  2. Related storiesAnthropic launches Claude Fable 5.2 and Claude Mythos 5.2
  3. Related storiesEnglish translation is being prepared.
  4. Related storiesEnglish translation is being prepared.