Understand this story
On October 20, 2026, AWS introduced a new bidirectional KV cache transfer mechanism in vLLM: in subsequent turns, the decode node sends the already-computed KV cache back to the prefill node, avoiding recomputation of the conversation history. On a 1P1D deployment on Amazon EC2 P5en, this mechanism improves TTFT for Qwen3-32B by up to 1.6x and for zai-org/GLM-5-FP8 by up to 3.3x. The capability was initially validated on AWS Trainium clusters and contributed to vLLM as an accelerator-agnostic feature that requires coordination with a KV-cache-aware router for cross-turn reuse. There are currently no further developments.
AI-generated from public reports
Latest development · Report
How vLLM's bidirectional KV transfer improves TTFT for multi-turn agent workloadsDevelopments
1 confirmed developments
AWS introduces vLLM bidirectional KV transfer to reduce multi-turn TTFT
Source:AWS Machine Learning
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Oct 20
AWS Machine LearningFirst-party
How vLLM's bidirectional KV transfer improves TTFT for multi-turn agent workloads
vLLM adds bidirectional KV transfer, allowing decode nodes to send already-computed KV caches back to prefill nodes in subsequent turns, avoiding recomputation of conversation history. On a 1P1D deployment on Amazon EC2 P5en, this mechanism improves TTFT by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8. This capability was initially validated on AWS Trainium clusters and contributed to vLLM; as an accelerator-agnostic feature, it requires coordination with a KV-cache-aware router for cross-turn reuse.