AI News

New requests still run old weights after vLLM reloads LoRA in place

UCSD Hao AI LabUpdated Nov 14 03:27

AI summary

Hao AI Lab used the vstack agent for formal verification and found that after vLLM reloads a LoRA adapter in place, new requests may still load the old weights, while vLLM reports the new version. The reason is that the reload keeps the adapter name and ID but points to the new file; requests already admitted before the reload still point to the old file, and when the number of adapters in vLLM's memory exceeds the limit and a reload is needed, the file is taken from that old request. The fix is to track the latest successful reload of each adapter and ensure loading from the new file.

Body

(5/7) Stale adapter: after a LoRA adapter is reloaded in place, new requests can still run on the old weights.

This breaks adapter updates: once a reload reports success, new requests for that adapter must use the new weights.

An in-place reload keeps the adapter's name and ID but points them to new files. A request admitted before the reload still points to the old files. So when vLLM serves more adapters than it keeps in memory and must load the adapter again, it can load it from that old request, so new requests run on the old weights while vLLM reports the new version.

An in-place LoRA reload while serving left new requests on old weights. Tracking each adapter's newest successful reload ensures vLLM loads from it, so old requests no longer bring back old weights.

PR: https://t.co/Cq8ktHgcGk

Source

UCSD Hao AI Lab