AWS Hosts NVIDIA Dynamo in SageMaker HyperPod Inference
On November 17, 2026, AWS announced that SageMaker HyperPod Inference supports NVIDIA Dynamo for hosted multi-node and disaggregated model serving. In tests with Qwen3-235B-A22B-FP8, the disaggregated deployment reached 207 output tokens/second per GPU at concurrency 80, higher than the 101 achieved by the aggregated deployment at concurrency 32. HyperPod provides EFA networking, GPU scheduling, health monitoring, and node replacement, and users can take the managed path via the dynamoSpec of InferenceEndpointConfig.
Why read this
AI score 57
The original text provides the managed configuration path and comparative data under latency constraints, which can help assess deployment trade-offs for multi-node inference.
Topics
Body
Amazon Web Services announced on November 17, 2026, that Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo for managed multi-node and disaggregated model serving, alongside benchmark results showing 2.05 times the highest qualifying aggregated output-token throughput per GPU under AWS’s latency targets.
The addition addresses two constraints in serving large language models: fitting model weights and the reusable key-value (KV) cache into GPU memory, and keeping long prompt processing from slowing token generation when both share GPUs. AWS illustrates the memory pressure with GLM-5.3, whose 8-bit floating-point (FP8) weights alone exceed the 640 GB of GPU memory on one ml.p5.48xlarge instance before KV cache and runtime overhead are included.
HyperPod supplies the managed infrastructure around the framework: Elastic Fabric Adapter (EFA) networking, coordinated GPU scheduling, health monitoring for nodes, GPUs and EFA, and node replacement. HyperPod manages that replacement while the orchestration components recreate affected workers.
Dynamo is an open-source distributed inference framework that works with vLLM, SGLang and TensorRT-LLM. Its disaggregated topology uses a frontend to route requests to prefill workers, which process prompts, and decode workers, which generate tokens. Those pools can scale and be tuned independently, but both load the full model weights; KV cache moves between them over EFA.
Aggregated serving runs both stages in the same worker. In either topology, workers can run on one instance or span multiple instances using tensor and pipeline parallelism to divide the model across GPUs. Dynamo’s KV-aware router considers reusable prefix cache and active worker load. AWS says prefix reuse can reduce prompt recomputation and improve time to first token, while load awareness helps avoid overloading individual workers.
Benchmarks With Latency Constraints
The Qwen3-235B-A22B-FP8 comparison used Dynamo and TensorRT-LLM on two p5en.48xlarge instances, totaling 16 H200 GPUs. AIPerf generated requests containing 4,000 input tokens and 200 output tokens. AWS retained only measured points with 90th-percentile time to first token below one second and median generation speed of at least 50 tokens per second per user.
AWS reported that disaggregated serving reached 207 output tokens per second per GPU at concurrency 80, compared with 101 at concurrency 32 for the highest qualifying aggregated result. The comparison therefore applies to latency-constrained throughput at different concurrency levels, rather than an unrestricted throughput maximum for either topology.
A separate Qwen3.5-397B-A17B-FP8 test used Dynamo and vLLM to compare eight H200 GPUs on one p5en.48xlarge instance with four GPUs on each of two instances, keeping the model, parallelism and workload configuration the same. AWS reported 126.4 output tokens per second per GPU for the two-node setup, or 99.1% of single-node throughput, at concurrency 32 with high prefix reuse. The model fits on one instance and would not normally be split in practice, so the test served as a controlled baseline for the throughput effect of crossing a node boundary.
Managed Configuration and Resource Planning
Users have two configuration paths. Existing Dynamo users can bring a DynamoGraphDeployment manifest describing the serving graph and manage storage, networking, pod placement, endpoints, TLS, registration and autoscaling themselves. The InferenceEndpointConfig path uses dynamoSpec for managed endpoint setup and Kubernetes Event-driven Autoscaling (KEDA) for either serving topology. It defaults to vLLM, provides defaults for KV-aware routing and replica counts, and supports an OpenAI-compatible interface through SageMaker endpoints.
The add-on installs Dynamo’s custom resource definitions and serving and scheduling components, while AWS Deep Learning Containers provide the runtime. On the managed path, the HyperPod Inference Operator stages weights from Amazon S3 or FSx and generates the DynamoGraphDeployment. Grove coordinates related pod groups, while KAI Scheduler uses gang scheduling so each multi-node worker’s pods are placed together. Adopting a supported Dynamo release requires updating the add-on.
AWS’s GLM-5.3 example makes the GPU-allocation choice concrete. Its aggregated configuration runs one model-worker replica across two ml.p5.48xlarge instances with 16-way tensor parallelism and two frontend replicas. The disaggregated variant configures one prefill replica and two decode replicas, each spanning two instances, for six instances in total. The separate replica counts show how a deployment can devote different amounts of capacity to prompt processing and token generation instead of scaling a single combined pool.
AWS recommends verifying EFA communication before benchmarking, testing representative prompt lengths, output lengths and prefix reuse, sweeping concurrency against service-level targets, and tuning topology and worker allocation from measured bottlenecks before changing engine settings.
HyperPod now handles more of the coordination and recovery work, but the benchmark conditions and deployment examples show that managed operation does not remove the serving trade-offs. The useful configuration still depends on measured prompt and generation behavior, the GPU allocation between worker pools, and the latency and cost priorities of the workload.
AI Trends