AI Trends

Trending story

Amazon SageMaker AI real-time endpoints support KV cache-aware routing

1 reports1 story updatesUpdated Nov 18 05:11

Understand this story

AI overview

Amazon SageMaker AI introduces KV cache-aware routing for real-time endpoints. Based on the cache status reported by the model container, it sends requests to instances that have cached a longer matching prefix and are not overloaded. To enable this policy, add the SM_VLLM_KV_EVENTS_CONFIG environment variable to the model to report cache events, and set RoutingStrategy to KV_AWARE in the endpoint configuration; client API calls remain unchanged. This policy requires at least two instances or two inference component copies, and falls back to load-aware routing when no matching instance is available.

AI-generated from public reports

Latest development · Report

Amazon SageMaker AI real-time endpoints support KV cache-aware routing

Developments

1 confirmed developments

  1. Amazon SageMaker AI real-time endpoints support KV cache-aware routing

    Source:AWS Machine Learning

Reports and discussion timeline

Browse reports and discussion by time. Their evidence levels remain distinct.

Nov 18

  1. AWS Machine LearningFirst-party

    Amazon SageMaker AI real-time endpoints support KV cache-aware routing

    Amazon SageMaker AI introduces KV cache-aware routing (KV-aware routing) for real-time endpoints, which, based on the cache state reported by the model container, routes requests to instances that have cached a longer matching prefix and are not overloaded. To enable it, add the SM_VLLM_KV_EVENTS_CONFIG environment variable to the model to report cache events, and set RoutingStrategy to KV_AWARE in the endpoint configuration; client calls to the API remain unchanged. This policy requires at least two instances or two inference component copies, and falls back to load-aware routing when no available match exists.

Related stories

  1. Related storiesNVIDIA Vera Rubin NVL72 Makes Its Debut in MLPerf Training v6.1
  2. Related storiesGoogle launches Gemini 4 Hydrogen on Antigravity
  3. Related storiesGoogle announces that Pathways is now generally available on Google Cloud Platform
  4. Related storiesNVIDIA introduces the five co-design elements of Vera Rubin NVL72