AI News

Hao AI Lab uses vstack to perform agentic formal verification on vLLM, finding more than 17 bugs

UCSD Hao AI LabUpdated Nov 14 03:26

AI summary

Hao AI Lab used vstack to perform agentic formal verification on vLLM and has found 17 bugs (and counting), publicly disclosing 5 of them along with the corresponding fix PRs. The issues include hybrid models continuing to generate from all-zero cache blocks, a small number of failed requests under the queue limit causing the server to continuously return busy, out-of-bounds writes to shared cache blocks in long streaming sessions, new requests still using old weights after a LoRA adapter is reloaded in place, and different images hitting the same cache key when two servers share an external KV cache. The relevant fixes are being submitted upstream, and the team says it will continue to work with the vLLM team and the open source community.

Why read this

AI score 58

The original article lists five categories of defects in vLLM that are difficult to find manually, along with the corresponding fix PRs, offering insight into the practical use of formal verification on inference engines.

Topics

Tags

Body

(7/7) Our fixes are going upstream now, and we're continuing to work with the vLLM team and the open-source community to bring more verifiable AI infra.

Huge thanks to the @vllm_project team for building such a robust engine and for their valuable feedback!

vLLM verification work by @AlexZhang152005 and @Junda_Su08 , with special thanks to @Yuxuan_Zhang13 , @yisun0618 , @Junda_Chen_ and @FuYichao123 for building this together at @haoailab !

Source

UCSD Hao AI Lab