NVIDIA 在 GB300 NVL72 上部署 Qwen3.8-2.4T-A95B，单 GPU 吞吐超 4K tokens/s

Technical Blog

sessionStorage.removeItem('facets.search.text'); sessionStorage.removeItem('facets.search.text-copy'); sessionStorage.removeItem('facets.search.filters');

Serve Qwen3.8-2.4T-A95B, a 2.4T-Parameter Model, with Configurable Reasoning on NVIDIA GB300 NVL72

Aug 12, 2026

Alibaba released the open weights for Qwen3.8-2.4T-A95B , a 2.4 trillion parameter mixture-of-experts model with 95 billion activated parameters per token, a hybrid full and linear attention architecture, and a context window of up to one million tokens. The model achieves over 4,000 tokens per second per GPU and over 350 tokens per second per user on NVIDIA GB300 NVL72 in FP8 precision on Day 0, with further gains expected from NVFP4 optimizations. NVIDIA NeMo AutoModel supports post-training the model for domain-specific use cases with full supervised fine-tuning or memory-efficient LoRA fine-tuning directly on Hugging Face checkpoints. Open-source inference recipes are available for SGLang, vLLM, and NVIDIA Dynamo, and the model can also be deployed via a model-free NVIDIA NIM container from NVIDIA NGC. Next Steps Try the model on DeepInfra for immediate hosted inference access. Download weights from Hugging Face to run the model locally. Deploy with a model-free NVIDIA NIM from NVIDIA NGC for production serving.

Powered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. Learn more

Alibaba released the open weights for Qwen3.8-2.4T-A95B (Qwen3.8-Max), its largest open-weight model, bringing near-frontier capabilities to the open ecosystem. It has 2.4T total parameters with 95B activated per token. It’s a fine-grained mixture of experts (MoE) architecture with a hybrid of full and linear attention, a context window of up to one million tokens, and an output length of up to 128K, designed for demanding reasoning and agentic workloads.

Deploying a 2.4T parameter open-weight model requires data-center-scale accelerated compute. Inference at this scale depends on extreme co-design across chips, system architecture, and software. NVIDIA is working with the open-source ecosystem to bring the model to multinode deployments through optimized kernels, inference runtimes, and distributed serving recipes.

Without additional model tuning, the model achieves a throughput of over 4K tokens per second per GPU and over 350 tokens per second per user on NVIDIA GB300 NVL72 in FP8 precision on Day 0. Further optimizations, including NVFP4 precision, are expected to deliver enhanced performance gains over time.

Qwen3.8-2.4T-A95B is built for the hardest agentic workloads like coding, large-scale document analysis, and long-running multi-step workflows. Unlike chat-first models that send a single prompt and receive a single reply, agentic applications accumulate system instructions, tool outputs, retrieved documents, code, logs, and multi-step reasoning traces across a workflow. As context grows, attention, compute, and KV cache memory become the binding constraints.

The full-attention and linear-attention hybrid architecture addresses this, and the model alternates between the two. In the full-attention layers, every token attends to every other token, and in the linear-attention layers, the growing KV cache is replaced with a bounded recurrent state. Qwen3.8-2.4T-A95B keeps both compute and memory bounded as context scales to up to one million tokens.

Fine-grained MoE makes the 2.4T parameter count practical to serve. Instead of a small number of large experts, capacity is distributed across a larger population of smaller experts, improving specialization and routing efficiency per unit of activated compute. A learned router activates only the experts needed per token, so serving costs track active parameters, not the full 2.4T parameters, delivering frontier-scale capacity at a fraction of the cost of a comparable dense model.

Built-in reasoning controls (low/high/xhigh) enable developers to configure inference depth per request, trading compute for reasoning quality depending on the task: dial up for complex multi-step reasoning or dial down for high-throughput document processing.

Figure 1. Overview of the Qwen3.8-2.4T-A95B linear gated delta networks plus full attention with fine-grained MoE architecture

Qwen3.8-2.4T-A95B optimized performance on GB300 NVL72

The GB300 NVL72 features a rack-scale architecture that integrates 72 NVIDIA Blackwell Ultra GPUs into a single platform. Its large, 72-GPU NVIDIA NVLink domain enables efficient all-to-all communication at 130 TB/s, eliminating bottlenecks that appear when expert traffic must cross traditional off-the-shelf networks.

Out of the box, Qwen3.8 2.4T-A95B running on NVIDIA Blackwell GB300 NVL72 delivers over 4K tokens per second per GPU and over 350 tokens per second per user, enabling AI factories to run large-parameter models in production at high throughput and low latency.

Figure 2. A Pareto curve showing Qwen3.8-2.4T-A95B achieving over 4K tokens per second per GPU at peak throughput on NVIDIA GB300 NVL72

Post-train Qwen3.8-2.4T-A95B and choose a serving path

NVIDIA supports multiple inference stacks to meet a variety of developer needs. SGLang , vLLM , and NVIDIA Dynamo provide open-source inference recipes for developers who require greater control over performance on the NVIDIA-accelerated platform.

It’s also available to deploy via a model-free NVIDIA NIM , a single inference container that serves any supported model. Download the model weights and deploy on Day-0 to serve fine-tuned checkpoints, and scale to production.

Developers can post-train the model for domain-specific use cases using NVIDIA NeMo AutoModel , a PyTorch-native fine-tuning library with Day-0 Hugging Face checkpoint support. Train directly on existing checkpoints without model conversion, with support for full SFT or memory-efficient LoRA fine-tuning.

Try out the model from these hosted APIs: DeepInfra , DigitalOcean, Fireworks AI, Modal , and OpenRouter

Download Qwen3.8-2.4T-A95B model weights from Hugging Face or ModelScope and deploy with a model-free NVIDIA NIM from NVIDIA NGC.