NVIDIA AIPerf：取代 GenAI-Perf 的大规模 LLM 推理基准测试工具

Technical Blog

sessionStorage.removeItem('facets.search.text'); sessionStorage.removeItem('facets.search.text-copy'); sessionStorage.removeItem('facets.search.filters');

Related Resources

Developer Tools & Techniques

English 中文

Sep 18, 2026

NVIDIA AIPerf replaces GenAI-Perf with a multiprocess architecture that prevents the client from becoming a bottleneck during high-concurrency LLM benchmarking. The tool supports over 15 endpoint types and public datasets including ShareGPT and trace replay formats from Mooncake, Baseten, and WEKA AgentX for realistic workload testing. Configurable arrival patterns such as constant, Poisson, and gamma distributions with tunable burstiness let engineers shape load to match production traffic characteristics. Core metrics including TTFT, ITL, request latency, and output token throughput are reported with percentile breakdowns and GPU telemetry when DCGM or pynvml is available. Next Steps Explore the AIPerf tutorials to learn advanced benchmarking scenarios. Review the Metrics Reference for full definitions of every reported measurement. Read How NVIDIA Dynamo 1.0 Powers Multi-Node Inference at Production Scale for distributed inference benchmarking context.

Powered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. Learn more

You’re deploying a model on a system. It starts up, prompts are getting responses. Now the hard question: Is this fast?

Your instincts might lead you to send curl commands, hand-roll an asyncio script, or vibe code yet another one-off load generator. All of these paths have the same problem: single-process performance limits, Python’s GIL capping concurrency, or numbers measured against a reference you built yourself. Either way, you end up with results you can’t fully trust, attached to tooling you’ll have to rewrite the moment requirements change.

What you need is a load client that can saturate a real server without becoming the bottleneck, produce output you can act on, and take five minutes to configure, not five hours. That’s NVIDIA AIPerf.

AIPerf is the designated successor to GenAI-Perf and is a ground-up rewrite. The design choices reflect hard lessons from running LLM benchmarks at scale:

For this walkthrough we’ll use Qwen3-0.6B served through vLLM. The model choice is deliberate; it’s small enough to run on a single GPU and fast enough to iterate on without waiting. The point isn’t to benchmark Qwen3-0.6B specifically; it’s to establish the measurement loop. Once you have that, swapping in a different model or endpoint is a one-flag change.

Start the Server

Pull and start vLLM with the reasoning parser enabled:

docker pull vllm/vllm-openai:latest docker run --gpus all -p 8000:8000 -e HF_TOKEN vllm/vllm-openai:latest \ --model Qwen/Qwen3-0.6B \ --reasoning-parser qwen3 \ --host 0.0.0.0 --port 8000

Install AIPerf

We can use uv to install a central copy:

uv tool install aiperf

Or for a virtual environment:

uv venv venv source venv/bin/activate uv pip install aiperf

One platform note: on aarch64, the crick dependency ships as source-only and requires a C toolchain ( build-essential on Debian/Ubuntu, Development Tools on RHEL). If the install stalls on that package, that’s why.

With the server up and AIPerf installed, we can now run our first profile:

aiperf profile \ --model Qwen/Qwen3-0.6B \ --endpoint-type chat \ --streaming \ --url localhost:8000 \ --synthetic-input-tokens-mean 128 \ --synthetic-input-tokens-stddev 0 \ --output-tokens-mean 128 \ --output-tokens-stddev 0 \ --extra-inputs min_tokens:128 \ --extra-inputs ignore_eos:true

A few flags here are doing more work than they look like:

--synthetic-input-tokens-stddev 0 and --output-tokens-stddev 0 pin the workload to exactly 128 input and 128 output tokens per request. This reproduces a commonly used static benchmark that holds request and output lengths constant.

--extra-inputs min_tokens:128 and --extra-inputs ignore_eos:true tell the model to actually emit 128 tokens rather than stopping early. Without these, the output token count is a suggestion. The model stops whenever it naturally finishes, which can be well short of your target OSL. Throughput numbers end up lower than they should be, and they’re not reproducible across runs.

--streaming is not optional if you want to measure TTFT and ITL. Without streaming, the server batches the full response before sending it, and there are no first- or decode-token events to measure.

Figure 1. An example animation of the AIPerf live dashboard user interface. The live dashboard shows the progress of the run, a listing of metrics along with their distributions, as well as a running log of events from the AIPerf backend

We’ll walk through how to read these numbers in the next section. For now, notice the shape of the output in Figure 2, below: latency broken down by percentile, throughput in tokens per second, and request-level statistics all in one place. That’s the baseline you’ll be comparing everything else against.

Figure 2. An example screenshot of the output metrics at the end of an AIPerf run which includes a summary of effective, active, summary statistics for a variety of different metrics along with percentile breakdowns for quick review, reproduction command line, and output locations

Reading the numbers: What AIPerf surfaces

Once a run completes, AIPerf prints a metrics table to the console and writes the full results to CSV and JSON. Here’s what you’re looking at.

The core four:

For full definitions of these and every other metric AIPerf reports, see the Metrics Reference .

Getting the full picture. Each of the above is reported in percentile breakdowns (p25, p50, p75, p90, p95, p99) alongside their minimums, maximums, averages, and standard deviations. These breakdowns matter because they can highlight long tail distributions; a server with a healthy mean TTFT and an outlier p99 looks fine in aggregate and fails in production.

Beyond the core four. With DCGM or pynvml available, AIPerf also pulls GPU power draw, utilization, and memory consumption into the same run output. Correlating a latency spike with a memory pressure event doesn’t require a separate profiling session, the telemetry is already there.

Now that our feet are wet with a static benchmark, we can start exploring something more dynamic. The section above provided an extremely fixed traffic pattern, but real inference traffic doesn’t follow a static pattern. To benchmark with a scenario that’s less rigid, we can use some of AIPerf’s synthetic workload knobs to introduce variability to our requests.

aiperf profile \ --model Qwen/Qwen3-0.6B \ --endpoint-type chat \ --streaming \ --url localhost:8000 \ --request-rate 10 \ --arrival-pattern poisson \ --synthetic-input-tokens-mean 512 \ --synthetic-input-tokens-stddev 128 \ --output-tokens-mean 128 \ --output-tokens-stddev 32 \ --random-seed 42 \ --request-count 200

A few things changed from the static benchmark above.

--arrival-pattern poisson with --request-rate 10 means requests arrive at an average of 10 per second, with inter-arrival times drawn from an exponential distribution. The server now experiences bursts and gaps rather than a single user stream, which is what queuing actually looks like under real traffic.

--synthetic-input-tokens-stddev 128 introduces variance around the 512-token mean, producing a mix of short and long prompts. The server has to handle variable prompt lengths during prefill rather than identical ones.

--output-tokens-stddev 32 adds variance on the output side. Notice that min_tokens and ignore_eos are gone from this command. In the static benchmark those flags pinned outputs to exactly 128 tokens to keep the baseline clean; we’re deliberately releasing that constraint so the output distribution can vary.

--random-seed 42 makes the Poisson timing and synthetic length draws reproducible. Rerunning this command produces the same sequence of requests.

--streaming is not optional. Without streaming, the server batches the full response before sending it, and there’s no first- or decode-token events to measure.

Looking at the LLM metrics from this run, the distributions are noticeably wider than the static baseline — which is expected when more requests are simultaneously competing for GPU access and prefill lengths vary per request.

Figure 3. An example screenshot of the summary statistics from the Poisson arrival pattern run. The distribution of statistics drastically differs from the 512/128 static scenario due to the new traffic pattern

Looking at the graphs in Figure 4, below, you can see that the Poisson command line introduced a request rate centered, but not exactly matching, around 10 requests/second. This arrival rate emulates jitter around when requests arrive compared to the constant mode which guarantees a fixed 10 requests/second.

Figure 4. The reported delay from first request dispatch, compared to the constant 10 requests/sec mode, showing variation in dispatch timing centered around the specified request rate

You can see in Figure 5, below, that there is a variation in the request length centered around the mean of 512 tokens, with input sequence lengths ranging 154 to 818 tokens.

Figure 5. A histogram showing the distribution of input (request) lengths centered around the requested 512 average token count

Comparing TTFT between the two runs, you can see that the Poisson run shows a much wider spread. More requests are simultaneously competing for GPU access, prefill lengths vary, and prefill and decode operations overlap. The single-concurrency case is an idealized scenario which runs one request at a time presenting the lowest possible TTFT, at the cost of throughput.

Figure 6. A histogram comparing the difference in time-to-first-token distribution between a single active user and Poisson arrival pattern AIPerf runs

In Figure 6, above, you can see that the single user run experiences less TTFT variability than the much more varied workload in the Poisson experiment.

This walkthrough covers the basics, but AIPerf is also built for more complex scenarios. The same tool handles multi-node Kubernetes deployments , KV cache reuse warm up mechanics , trace replay from production traffic, prefix synthesis, custom datasets, and sweep configurations across concurrency levels. If you’re running distributed inference at scale, see How NVIDIA Dynamo 1.0 Powers Multi-Node Inference at Production Scale .

The tutorials in the AIPerf repo are the fastest way in. The AIPerf repo and docs are the canonical reference for new features and contributions.

AIPerf is a collaborative effort between NVIDIA and external contributors. Thank you to the following: Loki Ravi, Dan Ferguson, and Sheng Moua (AWS) for the continual collaboration, cross-company validation, and efforts to standardize on AIPerf; Aaron Batilo (Coreweave) for the Weights & Biases exporter, acceptance-length spec-decode datasets, and hardening sweep/credit-dispatch reliability under concurrency; Shounak Ray (Baseten) for faithful Baseten trace replay support; Michael Feil (Baseten) for faster trace loading, and session affinity headers. Cristian Lopez (Pinterest) for his close collaboration on the DAG benchmarking methodology. We’re grateful to Ben Hamm for his product guidance while we designed, planned, and implemented AIPerf.