Hao AI Lab 用 vstack 在 vLLM 发现 17 个 bug
(3/7) Server lockout: with a queue limit set, a few failed requests make vLLM answer every new request with "busy" until a restart.
Hao AI Lab 用 vstack 智能体形式化验证在 vLLM 中发现 17 个(仍在增加)bug,并公开其中 5 个及修复 PR。设置队列上限时,2 个失败请求会让 vLLM 拒绝后续 2 个正常请求(2/2 次复现),数据并行下 8/8 新流量被路由到同一引擎;修复后失败请求归还队列槽位与路由条目,流量恢复 4/4 均分。其余问题包括混合模型从全零缓存块继续生成(GB200 上修复前 32/32 次命中全零)、流式长会话越界写共享缓存块、LoRA 原地重载后新请求仍用旧权重、多实例共享外部 KV 缓存时图像缓存键冲突。
正文
(3/7) Server lockout: with a queue limit set, a few failed requests make vLLM answer every new request with "busy" until a restart.
This breaks a basic availability promise: a request that fails must give back its place in the queue and in the load balancer.
When a queue limit is set, vLLM counts each request against it before handing it to the engine. If that hand-off fails, nothing returns the slot, so the client gets an error while its slot stays taken. Thus a few failed requests make the server refuse all work, though its health check still passes. (Without a limit, the leaked requests just keep using memory.) With data parallelism, the same failure also leaves a routing entry, so the load balancer keeps sending new traffic to one engine.
With a queue limit set, 2 failed requests made vLLM refuse the next 2 normal requests 2 of 2 times, while under data parallelism 8 of 8 went to one engine. Now a failed request returns its slot and routing entry, so both normal requests succeed and traffic splits 4 and 4 again. Our PR fixes both leaks.
PR: https://t.co/WmMSlKNYzr