Understand this story
Amazon SageMaker AI introduces KV cache-aware routing for real-time endpoints. Based on the cache status reported by the model container, it sends requests to instances that have cached a longer matching prefix and are not overloaded. To enable this policy, add the SM_VLLM_KV_EVENTS_CONFIG environment variable to the model to report cache events, and set RoutingStrategy to KV_AWARE in the endpoint configuration; client API calls remain unchanged. This policy requires at least two instances or two inference component copies, and falls back to load-aware routing when no matching instance is available.
AI-generated from public reports
Latest development · Report
Amazon SageMaker AI real-time endpoints support KV cache-aware routingDevelopments
1 confirmed developments
Amazon SageMaker AI real-time endpoints support KV cache-aware routing
Source:AWS Machine Learning
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 18
AWS Machine LearningFirst-party
Amazon SageMaker AI real-time endpoints support KV cache-aware routing
Amazon SageMaker AI introduces KV cache-aware routing (KV-aware routing) for real-time endpoints, which, based on the cache state reported by the model container, routes requests to instances that have cached a longer matching prefix and are not overloaded. To enable it, add the SM_VLLM_KV_EVENTS_CONFIG environment variable to the model to report cache events, and set RoutingStrategy to KV_AWARE in the endpoint configuration; client calls to the API remain unchanged. This policy requires at least two instances or two inference component copies, and falls back to load-aware routing when no available match exists.
AI Trends