AI News

Hao AI Lab used vstack to find 17 bugs in vLLM

UCSD Hao AI LabUpdated Nov 14 03:27

AI summary

Hao AI Lab used vstack agent-based formal verification to find 17 (and still increasing) bugs in vLLM, and publicly disclosed 5 of them along with fix PRs. When a queue limit is set, 2 failed requests cause vLLM to reject the next 2 normal requests (reproduced 2/2 times); under data parallelism, 8/8 new traffic is routed to the same engine; after the fix, failed requests return their queue slots and routing entries, and traffic distribution recovers to an even 4/4 split. The other issues include a hybrid model continuing to generate from all-zero cache blocks (32/32 hits on all-zero before the fix on GB200), out-of-bounds writes to shared cache blocks in long streaming sessions, new requests still using old weights after in-place LoRA reloading, and image cache key conflicts when multiple instances share an external KV cache.

Body

(3/7) Server lockout: with a queue limit set, a few failed requests make vLLM answer every new request with "busy" until a restart.

This breaks a basic availability promise: a request that fails must give back its place in the queue and in the load balancer.

When a queue limit is set, vLLM counts each request against it before handing it to the engine. If that hand-off fails, nothing returns the slot, so the client gets an error while its slot stays taken. Thus a few failed requests make the server refuse all work, though its health check still passes. (Without a limit, the leaked requests just keep using memory.) With data parallelism, the same failure also leaves a routing entry, so the load balancer keeps sending new traffic to one engine.

With a queue limit set, 2 failed requests made vLLM refuse the next 2 normal requests 2 of 2 times, while under data parallelism 8 of 8 went to one engine. Now a failed request returns its slot and routing entry, so both normal requests succeed and traffic splits 4 and 4 again. Our PR fixes both leaks.

PR: https://t.co/WmMSlKNYzr

Source

UCSD Hao AI Lab