NVIDIA 机密计算如何让 TensorRT LLM 在 Blackwell 上保持生产级推理性能

Technical Blog

sessionStorage.removeItem('facets.search.text'); sessionStorage.removeItem('facets.search.text-copy'); sessionStorage.removeItem('facets.search.filters');

Related Resources

Trustworthy AI / Cybersecurity

English 한국어 中文

NVIDIA Confidential Computing enables production AI inference on trusted hardware using memory-encrypted confidential virtual machines, confidential GPUs, and encrypted NVLink. TensorRT LLM applies CC-aware adaptations for secure data movement, kernel autotuning, and multi-GPU communication to preserve inference performance on NVIDIA Blackwell GPUs. In a controlled comparison on eight NVIDIA B200 GPUs with the DeepSeek-R1 model, confidential compute retained 96.1% to 98.2% of baseline output-token throughput while adding 1.2% to 4.3% per-token latency overhead across concurrency levels 1 through 16. Host-to-device transfers in B200 confidential computing pass through a software encrypted bounce buffer, so TensorRT LLM selects pageable memory instead of pinned memory and moves repeated token readback to an asynchronous worker. The kernel autotuner switches from CUDA events to the GPU %globaltimer for tactic measurements under confidential computing to avoid unstable timing signals. NVLS multicast is unavailable in B200 confidential configurations, so frameworks should detect NVLS availability and choose communication algorithms that minimize latency for the given message size and topology. Next Steps Review the NVIDIA Trusted Computing documentation to start planning a confidential inference deployment. Explore the latest TensorRT LLM features and release notes for CC-aware optimizations. Follow NVIDIA Confidential Compute news to stay updated on the latest developments.

Powered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. Learn more

As large language model (LLM) inference increasingly processes sensitive information and proprietary model context across personal, enterprise, and regulated settings, data must be processed inside a trusted environment. NVIDIA Confidential Computing (CC) provides a pathway for running these workloads securely using memory-encrypted confidential virtual machines (CVMs), confidential GPUs, and encrypted NVIDIA NVLink . This enables running production AI inference on trusted hardware.

Inference frameworks such as NVIDIA TensorRT LLM deliver best-in-class AI inference by combining framework-level optimizations with NVIDIA accelerated computing. However, when these frameworks run in a CC-enabled environment, secure execution changes assumptions behind memory movement, timing, scheduling, and multi-GPU communication. These changes introduce performance overhead if the runtime does not adapt. Maintaining high performance therefore requires the inference framework and the confidential computing environment to be optimized together.

For AI platform engineers evaluating confidential inference on NVIDIA Blackwell GPUs, this post examines the CC-aware adaptations that AI inference frameworks like TensorRT LLM use to account for secure execution while helping preserve inference performance. It presents a controlled methodology that teams can apply to quantify CC overhead on their own workloads.

Workload characteristics determine how visible CC overhead can be. High request volume can amortize fixed encryption costs by overlapping stalls with other work, making the direct effects harder to observe.

To expose these effects, select a workload with a long input context, extended output generation, and low concurrency. Long context stresses data movement during prefill, extended generation amplifies small per-token CC overhead during decode, and low concurrency limits the opportunity to hide those costs across concurrent requests.

The NVIDIA performance engineering team used these characteristics for the workload evaluated here.

To isolate the performance impact of CC, run the same workload under two conditions: confidential compute disabled (CC off) and confidential compute enabled (CC on) holding the model, hardware, framework version, sequence lengths, parallelism, and concurrency constant so that CC state is the only changing variable.

At each concurrency level:

Performance teams can use these measurements to quantify how much of the CC-off baseline is retained when CC is enabled for the target workload. The NVIDIA performance engineering team applied this comparison using the hardware and software configuration summarized in Table 2.

As shown in Figures 1 and 2, across concurrency 1–16, CC on retained 96.1– 98.2% of CC off output-token throughput, while mean TPOT remained within 1.2% to 4.3% of the baseline.

Figure 1. CC on output-token throughput relative to the CC off baseline at concurrency 1–16. CC on retained 96.1% to 98.2% of baseline throughput

Figure 2. Mean TPOT with CC enabled relative to the CC off baseline at concurrency 1–16. CC on introduced 1.2% to 4.3% TPOT overhead (lower TPOT is better)

Identifying and reducing CC overhead

The NVIDIA Blackwell confidential computing architecture introduces hardware-enforced security paths for protecting data and workloads in use. For a detailed overview of the architecture, see Hardware-Rooted AI Security That Won’t Slow You Down .

For TensorRT LLM users and framework developers, the following details show how these secure paths change common runtime assumptions and how TensorRT LLM adapts to reduce the resulting performance overhead.

In the B200 CC, host-to-device transfers pass through a software encrypted bounce buffer because the GPU cannot directly access protected CVM memory. This changes the behavior expected by inference frameworks: pinned memory no longer provides its usual asynchronous-transfer advantage, and some copies can block the calling thread.

The kernel autotuner normally uses CUDA events to compare candidate tactics. In the tested CC configuration, CUDA-event timestamps produced an unstable timing signal, which could cause the autotuner to select a slower tactic.

NVLS (NVLink SHARP) multicast is not available in B200 CC configurations. Without NVLS, NCCL_SYMMETRIC cannot provide its intended multicast benefit but may still incur memory registration and cross-rank synchronization costs before using a non-multicast collective path.

NVIDIA Confidential Computing extends hardware-enforced protection across confidential VMs, NVIDIA Blackwell GPUs, and encrypted NVLink, protecting proprietary models, enterprise context, and sensitive prompts while they are processed.

Confidential computing does not remove the need for performance engineering—it makes framework awareness even more important. With TensorRT LLM CC-aware adaptations to secure data movement, autotuning, and multi-GPU communication in place, confidential DeepSeek-R1 inference retained more than 96% of CC-off output-token throughput while keeping per-token latency overhead below 5% on eight NVIDIA B200 GPUs.

As organizations move private inference into production, security configuration and inference optimization should be approached as a single deployment problem. Enable confidential computing, attest the environment, and benchmark CC-on and CC-off using the exact workload you intend to serve. For AI platform engineers and TensorRT LLM users moving private inference into production, security configuration and inference optimization should be treated as a full-stack engineering effort.

To start planning your confidential inference deployment, use the NVIDIA Trusted Computing documentation and explore the latest TensorRT LLM features and release notes . To stay up to date with the latest developments, follow NVIDIA Confidential Compute news .

I would like to thank Dan Hansen, Sheel Pethe, Samuel Mendoza-Jonas, Moein Ghaniyoun, Vidhya Krishnan, Avinash Ahuja, Laikh Tewari, Laura Martinez, and Matheen Raza for their engineering contributions, technical guidance, analysis, and thoughtful review throughout this work.