AI Trends

Trending story Continuously updated

AWS adds managed NVIDIA Dynamo support to SageMaker HyperPod Inference

2 reports2 story updatesUpdated Nov 18 07:51

Understand this story

AI overview

On November 17, 2026, Amazon Web Services announced that Amazon SageMaker HyperPod Inference supports managed NVIDIA Dynamo, combining Dynamo's distributed inference capabilities with HyperPod's infrastructure management. AWS Machine Learning explained that the support covers multi-node deployment, prefill/decode disaggregation, and KV-aware routing; taking GLM-5.3 as an example, its FP8 weights exceed the 640 GB of GPU memory on a single ml.p5.48xlarge, requiring cross-node deployment. Deployment can optionally use dynamoSpec with a bring-your-own DGD or InferenceEndpointConfig, and the inference engine supports vLLM, SGLang, and TensorRT-LLM, with vLLM as the default. Unite.AI reported the same day that in a Qwen3-235B-A22B-FP8 test, disaggregated deployment reached 207 output tokens per second per GPU at a concurrency of 80, higher than the 101 achieved by aggregated deployment at a concurrency of 32. The support is currently in a generally available state following the announcement.

AI-generated from public reports

Latest development · Report

AWS Hosts NVIDIA Dynamo in SageMaker HyperPod Inference

Developments

1 confirmed developments

  1. AWS adds managed NVIDIA Dynamo support to SageMaker HyperPod Inference

    Source:AWS Machine Learning

Reports and discussion timeline

Browse reports and discussion by time. Their evidence levels remain distinct.

Nov 18

  1. Unite.AI

    AWS Hosts NVIDIA Dynamo in SageMaker HyperPod Inference

    On November 17, 2026, AWS announced that SageMaker HyperPod Inference supports NVIDIA Dynamo for hosted multi-node and disaggregated model serving. In tests with Qwen3-235B-A22B-FP8, the disaggregated deployment reached 207 output tokens/second per GPU at concurrency 80, higher than the 101 achieved by the aggregated deployment at concurrency 32. HyperPod provides EFA networking, GPU scheduling, health monitoring, and node replacement, and users can take the managed path via the dynamoSpec of InferenceEndpointConfig.

  2. AWS Machine LearningFirst-party

    Amazon SageMaker HyperPod Inference now hosts NVIDIA Dynamo

    Amazon SageMaker HyperPod Inference now supports hosting NVIDIA Dynamo, combining Dynamo's distributed inference capabilities with HyperPod's infrastructure management, and supporting multi-node deployment, prefill/decode disaggregation, and KV-aware routing. Taking GLM-5.3 as an example, its FP8 weights exceed the 640 GB GPU memory of a single ml.p5.48xlarge, requiring cross-node deployment; for deployment, you can optionally use dynamoSpec with your own DGD or InferenceEndpointConfig, and the inference engine supports vLLM, SGLang, and TensorRT-LLM, with vLLM as the default.

Related stories

  1. Related storiesGoogle launches Gemini 4 Hydrogen on Antigravity
  2. Related storiesNVIDIA showcases full lineup of local AI hardware products at GTC Berlin
  3. Related storiesSGLang natively supports and powers the Beam production API
  4. Related storiesMeta releases open-weight model Muse Spark Open