Oct 22, 2026
By Arvind Gopalakrishnan, Eugene Rakhmatulin, Manini Sharma and Prerit Rodney
Discuss (0)
AI-Generated Summary
- NVIDIA DGX Spark now supports official eight-node clusters, scaling unified memory to 1 TB and enabling local execution of frontier-class models such as the full GLM 5.3 (753B) from Z.ai.
- A Spark-optimized collective backend called RoCEnante reduces all-reduce latency for tensor-parallel decode on switch-connected clusters, delivering a 4.43x decode speedup at eight nodes compared to 3.05x without it.
- Benchmark data shows decode throughput scaling from roughly 12 tokens per second on one node to 52 tokens per second on eight nodes using Qwen3.8-27B (NVFP4), with switch topologies enabling flexible workload partitioning and ring topologies offering cost-effective four-node inference.
- NVIDIA NeMo Switchyard routes requests between model pools on a shared eight-node cluster, matching Qwen-only quality on a tool-use benchmark while raising aggregate throughput up to 3x when most traffic shifts to a smaller model.
- Pre-training a nanochat model from scratch across eight nodes achieves a 7.79x speedup at 97.4% scaling efficiency, demonstrating near-linear training performance for small models on the desk.
Next Steps
- Explore Connect multiple DGX Spark machines for distributed workloads to set up a multi-node cluster.
- Serve LLMs with vLLM across DGX Spark for distributed inference.
- Review model compatibility guide and Switchyard orchestration resources for multi-model routing.
AI developers aren’t just running a single model anymore—they’re orchestrating multiple models in parallel, chaining inference calls, and experimenting with frontier-class architectures that demand hundreds of gigabytes of memory. NVIDIA DGX Spark was built to bring AI development to the desk, and now you’ll be able to link even more Spark nodes together.
At GTC Berlin, NVIDIA announced official support for eight-node DGX Spark clusters, scaling unified memory to 1 TB and opening the door to frontier-class models, multi-agent workloads, and near-linear pre-training—all running locally.
NVIDIA Sync makes it easy to cluster. Its Cluster Assistant handles setup and device management. Support for eight-node clusters in NVIDIA Sync is coming by the end of the year.
Read on for benchmark data on how inference, orchestration, and training scale as the cluster grows.
Run larger, newer frontier-class models
The biggest benefit of an eight-node cluster is memory. A single DGX Spark provides 128 GB of unified memory; four nodes reach 512 GB; eight nodes reach 1 TB. That capacity opens the door to a class of models that previously wouldn’t start on a four-node system, including the full GLM 5.3 (753B) model from Z.ai.
Beyond simply fitting these models, the cluster is designed to serve them interactively. As an early example, we ran GLM 5.3 Flash across an eight-node cluster, and measured roughly 80 tokens per second (TPS) in generation, suitable for interactive local development against a frontier-class mixture-of-experts model.
The full GLM 5.3 (753B) checkpoint runs across the 8x 128 GB cluster, making it one of the largest open models you can serve locally.
| Concurrency | Prefill, total tokens per second (TPS) | Prefill, per user (TPS) | Decode, total (TPS) | Decode, per user (TPS) |
| 1 | 1,600 | 1,600 | 31 | 31 |
| 4 | 1,600 | 750 | 51 | 21 |
| 8 | 1,600 | 473 | 67 | 18 |
Faster collectives with a Spark-optimized backend
Tensor-parallel inference spends a meaningful fraction of its time in collective operations such as all-reduce. On switch-connected clusters, we have been evaluating a Spark-optimized collective backend, RoCEnante, that reduces collective latency for the small-message, low-concurrency, modest-rank-count pattern typical of tensor-parallel decode on local AI hardware.
This backend implements an all-to-all exchange followed by local reduction. Each rank shares its input with every other rank, then adds the inputs locally. A single GPU kernel coordinates the exchange and reduction.
The main benefit is fewer communication steps. Ring all-reduce passes data between neighboring ranks over multiple rounds. This approach exchanges data directly with all peers, avoiding that chain of forwarding. For small messages and a modest number of ranks, this can reduce latency.
The tradeoff is more data movement. Each rank sends its full input to every peer and reduces all the inputs itself. As messages or number of ranks (nodes) grow, that extra traffic becomes more expensive, making ring all-reduce’s bandwidth efficiency increasingly attractive. If you need to support high concurrency, larger clusters, or ring topologies, NVIDIA Collective Communications Library (NCCL) would perform better.
In practice we see roughly 5% higher prefill and 21% higher decode throughput at four nodes (NVFP4), and about 18% higher decode throughput at eight nodes on the non-quantized model, consistent with the latency-versus-bandwidth tradeoff above. The backend is merged into FlashInfer and falls back transparently to NCCL if its startup test fails.
The backend applies only to switch-connected topologies, because it relies on direct point-to-point exchange between every node. It is not a replacement for NCCL, which remains the supported default and the better choice as cluster size and concurrency grow. The original one-shot all-reduce approach was contributed by the community (Jason Cook/el8).
Two, four, eight: Scaling the cluster
Low-latency NVIDIA ConnectX-7 NICs and RDMA on Converged Ethernet (RoCE) communication let DGX Spark nodes exchange tensors efficiently, so you can scale a model across more nodes to fit it in memory and to improve throughput. To isolate cluster scaling behavior and remove context-dependent behavior, we benchmarked a dense Qwen3.8-27B (NVFP4) model without speculative decoding/MTP across one to eight nodes:
| Number of Spark nodes | Topology | Prefill (TPS) | Prefill scaling | Decode (TPS) | Decode scaling |
| 1 | — | 2,635 | 1.00x | 11.86 | 1.00x |
| 2 | Direct connect | 3,285 | 1.25x | 21.71 | 1.838x |
| 4 | Physical ring | 4,446 | 1.69x | 32.82 | 2.77x |
| 4 | Switch (ROCEnante) | 4,556 | 1.73x | 36.34 | 3.06x |
| 8 | Physical ring | 4,962 | 1.88x | 40.11 | 3.38x |
| 8 | Switch (ROCEnante) | 5,105 | 1.94x | 52.58 | 4.43x |
Two patterns stand out. First, decode throughput scales strongly with node count, from about 12 TPS on one node to 52 TPS on eight nodes.
Second, the best topology depends on the workload: A physical ring delivers excellent decode throughput for a single model spread across all nodes. A switch is needed once you want to split the cluster into independent groups, or run workloads that require direct any-to-any connectivity between nodes.
To try this yourself, this networking guide walks through cluster setup starting from a two-node configuration and scaling up.
Switchless ring topology and setup simplicity
Not every cluster may need a switch. DGX Spark nodes can be wired directly together in a ring, which lowers cost and simplifies setup—an attractive option for enthusiasts and small deployments.
Within a four-node DGX Spark deployment, NCCL defaults to ring-based algorithms for tensor-parallel communication, allowing a switchless ring topology to align perfectly with traffic flows and offer impressive throughput at lower expense. As Table 2 shows, the four-node ring performs well in prefill and decode throughput for this workload.
The picture changes as you scale or specialize:
- Eight nodes or split allocations: If you want to split a cluster into, say, two groups of four, a ring cannot express that topology; you need a switch.
- All-to-all and traditional HPC workloads: Anything that depends on point-to-point patterns (for example, MPI-based clustering) requires a switch; a ring would pay a hop penalty.
- The optimized collective backend: The Spark-optimized backend described above requires direct point-to-point connectivity between every node, so it applies to switched configurations only.
In short: Reach for a switchless ring when you want the simplest, lowest-cost four-node inference box, and choose a switch when you need to scale up to eight nodes, split the cluster, or run communication patterns that a ring cannot serve.
Multi-model orchestration with Switchyard
Not every request needs your largest model. On a shared eight-node cluster, you can get substantially more aggregate throughput by routing easy requests to a small, fast model and reserving a larger model for the hard ones.
NVIDIA NeMo Switchyard makes this practical: A lightweight classifier inspects each request (or session) and routes it to the appropriate model pool, with a load balancer distributing traffic across replicas. Because this layout splits the eight nodes into independent model pools, Switchyard runs on a switch-connected cluster, which provides the any-to-any connectivity needed to route requests across replicas.
In our prototype, we used OpenBMB’s MiniCPM5 2B as the classifier and execution model for simple tasks and Qwen3.8 Flash Next for complex ones. On a standard-plus-hard tool-use benchmark, the routed agent matched the quality of the Qwen-only configuration while freeing capacity on the cluster.
On the 92-case tool-use benchmark (including hard cases), the routed agent scored 91/100, matching the Qwen-only configuration.
Because MiniCPM serves roughly an order of magnitude more requests per second than Qwen per replica, shifting part of the traffic to the smaller model raises the throughput ceiling of the whole cluster. Allocating three Qwen replicas (two Spark nodes each) and two MiniCPM replicas (one Spark each) across the eight nodes, we get:
- ~1.5x aggregate throughput over a Qwen-only baseline when half of requests route to Qwen.
- ~3.0x aggregate throughput when a quarter of requests route to Qwen and the rest to MiniCPM.
A routing policy is a simple YAML file, and NVIDIA provides a sample policy as a starting point; developers can tune the split and classifier to their own workload.
Pre-training from scratch: near-linear scaling with nanochat
Training behaves differently from inference. When each node holds a full copy of the model, the only cross-node communication is gradient synchronization once per step, so clustering scales close to linearly. To demonstrate, we pre-trained a nanochat model from scratch (about 896.5M parameters, 20 transformer layers, hidden dimension 1,280- and 2,048-token sequences) and measured training throughput as we added nodes.
| Nodes | TPS | Speedup | Scaling efficiency |
| 1 | 8,741 | 1.00x | 100.0% |
| 2 | 17,202 | 1.97x | 98.4% |
| 4 | 33,640 | 3.85x | 96.2% |
| 8 | 68,079 | 7.79x | 97.4% |
Eight nodes deliver a 7.79x speedup at 97.4% efficiency, the kind of near-linear scaling that makes an eight-node Spark cluster a credible platform for training small models from scratch, not just serving large ones. It is also a natural on-ramp workflow: Prototype and pre-train locally, then scale out as the model grows.
To reproduce the nanochat pre-training experiment or adapt it to your own model, the full code is available on GitHub.
Get started with multi-system DGX Spark
Eight-node DGX Spark clusters give you the memory to load frontier-class models, the flexibility to orchestrate several models at once, and the efficiency to train small models from scratch, all on the desk.
To learn more, check out:
- Connect multiple DGX Spark machines for distributed workloads
- Serve LLMs with vLLM across DGX Spark
- Explore the model compatibility guide and Switchyard orchestration resources
Start building on NVIDIA DGX Spark today.
Discuss (0)Tags
Agentic AI / Generative AI | Developer Tools & Techniques | General | NeMo | Intermediate Technical | Benchmark | AI Agent | AI Inference | ConnectX | DGX Spark | GTC Berlin 2026 | LLMsAbout the Authors
About Arvind Gopalakrishnan
Arvind Gopalakrishnan is director of systems software at NVIDIA, leading software engineering for GPU local AI platforms across NVIDIA DGX Spark and NVIDIA DGX Station. He focuses on software engineering on these platforms, spanning inference performance, multi-node interconnect architecture, model enablement, platform stability, certification, and release readiness — partnering closely with product, firmware, and hardware teams. With deep systems-level technical grounding spanning GPU driver, platform, and NVIDIA software stack, Arvind combines hands-on engineering leadership with cross-functional reach to align teams and drive execution across the stack. View all posts by Arvind Gopalakrishnan
About Eugene Rakhmatulin
Eugene Rakhmatulin is a principal software engineer at NVIDIA, focused on local AI solutions. He is the creator of spark-vllm-docker, an open source project that simplifies deploying and running large language models on NVIDIA DGX Spark using vLLM, supporting both standalone systems and clusters. With three decades of experience in software development, systems architecture, and technology leadership, he has built software platforms and led engineering teams across startups, nonprofit organizations, and multinational companies. View all posts by Eugene Rakhmatulin
About Manini Sharma
Manini Sharma is a senior product manager at NVIDIA, responsible for NVIDIA RTX Spark Desktops and DGX Spark. With more than two decades of experience spanning product management, product marketing, and software engineering, she has led flagship silicon and platform products from concept through market. She combines deep technical grounding with business strategy to drive product growth and deliver platforms that resonate with developers and customers alike. View all posts by Manini Sharma
About Prerit Rodney
Prerit Rodney is a senior software engineer at NVIDIA, focused on local AI systems and experience on NVIDIA DGX Spark, DGX Station, and RTX. He has a background in designing and developing operating systems and application software for different ecosystems. He has led several technical projects and improved system and operational efficiency across multiple domains. View all posts by Prerit Rodney