如何在 Amazon SageMaker AI 上用并发扫描为生成式 AI 端点做容量规划

Concurrency sweeps help you right-size a generative AI endpoint by finding the instance type and serving configuration that maximizes price-performance while holding latency within acceptable bounds. Without a systematic approach, right-sizing means deploying, load-testing manually, adjusting, and repeating until the numbers look acceptable. Choose five ml.g7e.2xlarge instances when one would suffice, and you burn your budget on idle GPUs. Choose too few, and requests queue, latency spikes, and users experience degraded service.

Concurrency sweeps address this problem. A concurrency sweep is a systematic benchmarking approach that sends controlled, increasing levels of concurrent traffic to your Amazon SageMaker AI endpoint and analyzes its performance. Concurrency sweeps are built into Amazon SageMaker AI Inference Recommendations , so there’s no custom load-testing infrastructure to build or maintain.

In this post, we walk through deploying the NVIDIA Nemotron-3 Nano 30B model, running automated concurrency sweeps, and using the results to make data-driven capacity decisions. By the end, you will know how many concurrent requests your endpoint can handle before latency becomes unacceptable, and how to automate that discovery.

A concurrency sweep sends a controlled number of simultaneous requests to your SageMaker AI endpoint and measures two metrics at each level:

By progressively increasing the concurrency (for example, 64 to 256, and then 1,024 simultaneous requests), you can trace a curve that reveals your endpoint’s saturation point . This is the point where adding more concurrent traffic stops improving throughput and degrades latency.

A concurrency sweep gives you three data points for production planning:

Let’s now look at how the end-to-end workflow comes together.

The concurrency sweep workflow has four steps:

The following diagram illustrates this process.

Figure 1: The four-step concurrency sweep workflow

The following four steps will take you from a fresh deployment to a complete capacity profile. Each step builds on the previous one, so we recommend following along with the accompanying notebook .

Before getting started, make sure that you have:

We deploy NVIDIA Nemotron-3 Nano 30B, a Mixture-of-Experts (MoE) model with only 3B active parameters, to an ml.g7e.2xlarge instance. This instance is backed by an NVIDIA Blackwell GPU , which provides a strong price-performance ratio for inference workloads.

We use the vLLM Deep Learning Container for SageMaker AI, configured through SM_VLLM_* environment variables:

VLLM_IMAGE = ( f"763104351884.dkr.ecr.{region}.amazonaws.com/" f"vllm:0.19.1-gpu-py312-cu129-ubuntu22.04-sagemaker" )

environment = { "SM_VLLM_MODEL": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16", "SM_VLLM_ENFORCE_EAGER": "true", # required for Mamba-Transformer hybrid "SM_VLLM_TENSOR_PARALLEL_SIZE": "1", # single GPU "SM_VLLM_GPU_MEMORY_UTILIZATION": "0.85", "SM_VLLM_MAX_MODEL_LEN": "10240", "SM_VLLM_ENABLE_PREFIX_CACHING": "true", "SM_VLLM_TRUST_REMOTE_CODE": "true", }

Three of these settings are worth explaining. The SM_VLLM_ENFORCE_EAGER flag is required because Nemotron-3 Nano uses a Mamba-Transformer hybrid architecture that requires eager execution mode. We set the GPU memory utilization to 0.85 to leave headroom for KV cache growth under high concurrency. Prefix caching is enabled to improve performance in scenarios where system prompts are repeated across requests to benefit from reusing cached key-value pairs.

After creating the model, endpoint configuration, and endpoint, we verify the deployment with a validation before moving on to benchmarking.

Before running any load test, the benchmark engine needs to know what kind of traffic to simulate. We define a workload profile that mirrors a realistic generative AI inference pattern using CreateAIWorkloadConfig . The following table summarizes the values we used for defining the workload. You can follow the code in the accompanying notebook .

The choice of 1,024 input tokens and 256 output tokens is representative of Retrieval Augmented Generation (RAG) or summarization workloads. If your application uses shorter prompts and longer completions (such as code generation), adjust these values accordingly. Streaming is enabled to capture time to first token (TTFT) metrics, which are important for interactive user experiences where perceived responsiveness matters as much as raw throughput.

With the workload profile defined, the next step is to launch the sweep.

We can now create the benchmark job with the CreateAIBenchmarkJob API with the parameters to use in the sweep. In the scenario illustrated in the sample notebook, we have:

sweep_params = { "concurrency": [64, 256, 1024], # powers of 4 "request_count": 1024, # requests per concurrency level }

The benchmark engine ( AIPerf ) runs each concurrency level sequentially within a single job, sending the total number of requests defined under request_count at each level. Running the levels sequentially instead of in parallel means each measurement reflects a clean, isolated load. This approach gives you statistically stable estimates while keeping costs contained.

When the job completes, the results are written to the Amazon S3 path defined under OutputConfig as a tarball containing per-level metrics in JSON format.

After the job completes, we can download and parse the output for analysis. Four metrics matter most when reading the results:

When you plot throughput against latency across the concurrency levels, the saturation point is visible as a “knee” in the curve: throughput flattens while p99 latency bends sharply upward. This happens at 256 concurrent requests in our test scenario. Concurrency levels below the knee are your safe operating region. Above it, the endpoint is overloaded and users are waiting.

Figure 2: Throughput and p99 latency across concurrency levels, with the saturation knee at 256 concurrent requests

At this point you have a complete picture of how your endpoint behaves under load. For many teams, this is sufficient to make a confident capacity decision. If you want to automate the search for the optimal operating point across model versions or instance types, the benchmark engine offers an automated alternative.

The concurrency sweep in Step 3 requires you to choose the concurrency levels to test. You might not know the right range, or you might want to automate capacity planning across model versions. In either case, replace the fixed concurrency list with a search recipe in the same CreateAIBenchmarkJob call. The max-concurrency-under-sla recipe accepts one or more SLA thresholds and searches for the highest concurrency that satisfies all of them.

The following table describes the available SLA threshold parameters:

For example, we can use the following search parameters to find the maximum concurrency where p99 end-to-end latency stays under 50 seconds:

search_params = { "search_recipe": "max-concurrency-under-sla", "e2e_sla_ms": 50000, "concurrency_min": 16, "concurrency_max": 4096, "search_max_iterations": 10, "search_initial_points": 512, "request_count": 1024, }

The search engine uses an optimization planner that can converge on the answer in fewer iterations than a linear sweep. It starts with a broad range and narrows progressively, evaluating only the concurrency levels needed to identify the boundary. This can reduce both the number of iterations and the total cost of the search.

In this example, the planner starts at concurrency 16 and doubles through each iteration. At concurrency 512, the first SLA violation occurs, either because the p99 end-to-end latency exceeded the threshold or because invocations failed. The planner then narrows the search to the 256–512 range and finds that concurrency 320 meets the SLA at 2,782 tokens per second. The search runs for at most the number of iterations you define in search_max_iterations .

A single SLA threshold is insufficient for most production workloads, as some scenarios require that a model must meet multiple SLAs at the same time. For instance, in interactive user experiences, you might need to control both the end-to-end latency and the time to first token. You can still use the max-concurrency-under-sla search recipe by passing multiple SLAs under search parameters:

search_params = { "search_recipe": "max-concurrency-under-sla", "e2e_sla_ms": 50000, # p99 end-to-end < 50s "ttft_sla_ms": 1500, # p95 time-to-first-token < 1.5s "concurrency_min": 16, "concurrency_max": 4096, "search_max_iterations": 10, "request_count": 1024, }

Iteration Concurrency Throughput (OTPS) Passed SLA?

In this case, we observe that when putting SLAs in both the end-to-end latency and time to first token, the maximum level of concurrency supported is 80.

With the benchmarking complete, let’s clean up the resources we created during this walkthrough.

To avoid ongoing charges, delete your SageMaker endpoints , endpoint configurations , and models after testing. Endpoints bill by the hour whether or not they’re receiving traffic.

Concurrency sweeps replace guesswork with data in the capacity planning process for generative AI endpoints. Instead of over-provisioning as a precaution or discovering bottlenecks in production, you can systematically map your endpoint’s performance envelope. You can then make informed decisions about fleet size before a single user request hits your system.

In this post, you learned how to:

Related resources:

Mona currently works as Sr AI/ML specialist Solutions Architect at Amazon. She is a published author of three books and her latest book is AI Agents on AWS. She has authored 20+ blogs on AI/ML and cloud technology and a co-author on a research paper on CORD19 Neural Search which won an award for Best Research Paper at the prestigious AAAI (Association for the Advancement of Artificial Intelligence) conference.

Hrushikesh is a Principal Solutions Architect for AI/ML startups with expertise in both AWS machine learning and networking services. He helps startups building generative AI, autonomous vehicles, and ML platforms to run their business efficiently and effectively on AWS.

Felipe is a Principal AI/ML Specialist Solutions Architect at AWS. Prior to joining AWS, Felipe worked with GE Digital and SLB, where he focused on modeling and optimization products for industrial applications.

Lokeshwaran is a Senior Deep Learning Compiler Engineer at AWS, specializing in ML optimization, model acceleration, and AI security. He focuses on enhancing efficiency, reducing costs, and building secure ecosystems to democratize AI technologies, making cutting-edge ML accessible and impactful across industries.

Sheng is a software engineer on SageMaker focused on inference and model optimization, building scalable, user-friendly tools that help customers deploy AI models faster and more efficiently.