NVIDIA Dynamo 推出影子引擎恢复，LLM 推理故障恢复从 283 秒缩短到 7.3 秒

Technical Blog

sessionStorage.removeItem('facets.search.text'); sessionStorage.removeItem('facets.search.text-copy'); sessionStorage.removeItem('facets.search.filters');

Related Resources

Developer Tools & Techniques

English 中文

Aug 25, 2026

Shadow engine recovery in NVIDIA Dynamo moves most recovery work off the serving path by keeping a fully initialized shadow engine idle on the same GPUs as the active engine. The GPU Memory Service shares existing weights between engines without creating another copy in HBM, allowing a shadow to take over within seconds if the active process fails. In a two-worker GLM-5.2 deployment on NVIDIA B200 nodes, shadow engine recovery resumed serving in 7.3 seconds compared to 283 seconds for a cold restart, nearly 39 times faster. The feature requires Dynamic Resource Allocation on Kubernetes 1.34 or newer with the NVIDIA GPU DRA driver installed, and vLLM is the primary supported backend. Next Steps Start with the Kubernetes quickstart to create a running deployment. Follow the Shadow Engine Recovery deployment workflow and use the vLLM failover example as a complete manifest. Visit the ai-dynamo/dynamo repository to ask questions, report issues, or contribute.

Powered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. Learn more

When an LLM engine process fails, the standard recovery path involves a cold restart. This requires loading weights into HBM from storage, compiling kernels, and capturing NVIDIA CUDA graphs. For large models, initialization can take several minutes, during which surviving workers must absorb the displaced traffic.

Shadow engine recovery, available as a preview feature in NVIDIA Dynamo, moves most of this recovery work off the serving path. It keeps a fully initialized shadow engine idle on the same GPUs as the active engine. The GPU Memory Service (GMS) shares the existing weights between the engines without creating another copy in HBM. If the active process fails, the shadow takes over within seconds. Re-initialization occurs in the background entirely off the serving path.

We measured the impact by deliberately terminating one worker in a two-worker GLM-5.2 deployment. Without shadow engine recovery, the remaining worker served all incoming traffic during the 283-second cold restart, increasing TTFT and reducing per-user decode rate throughout the outage. With shadow engine recovery, a second worker resumed serving in 7.3 seconds, nearly 39 times faster, minimizing disruption to service quality.

Figure 1. Time until a second worker resumes serving after one of two workers fails. A cold restart reloads weights, sizes the KV cache, autotunes, and recaptures CUDA graphs. A preinitialized shadow engine can begin serving much sooner

Why LLM inference recovery is slow: two core problems

Production LLM engines commonly experience recoverable software faults, including process crashes, recoverable CUDA errors, and transient collective failures. In these cases, the hardware, drivers, and node remain healthy; only the process holding the corrupted state is lost, and a replacement engine can typically start on the same GPUs.

So why can’t that fresh engine skip the initialization cost? Two problems stand in the way:

Shadow engine recovery addresses each problem with a targeted optimization: decoupling weight lifetime from the engine process, and completing non-transferable initialization before a failure.

Shadow engine recovery combines persistent GPU memory, a pre-warmed standby engine, and worker-level coordination to recover without a cold restart.

The GPU Memory Service (GMS) manages specific memory regions, such as weights, independently of the engine process. By using a process distinct from the engine to own these regions, weights remain resident in memory even as engines are restarted. As a result, a new engine on the same GPU can attach to existing memory.

GMS is a per-GPU sidecar that owns physical GPU memory on behalf of inference engines. It is mostly dormant and has no CUDA context of its own; it allocates physical pages, hands out handles to them, and arbitrates which engines may read or write at any time. Engines connect, import handles, and map the underlying pages at virtual addresses in their own CUDA contexts. Mapping happens once, at startup, and GMS is not involved in any subsequent access.

This functionality is built on the CUDA Virtual Memory Management API. With this API, physical GPU memory and its associated virtual addresses can have independent lifetimes. Since physical allocations are reference-counted, they survive as long as any process maintains a mapping. Two engines mapping the same weight tensor access the same physical bytes, each using virtual addresses local to its context. A kernel reading a weight dereferences an ordinary pointer into the same HBM the weight would have occupied anyway, so a GMS-backed read costs no more than an engine-allocated one.

Figure 2. Each engine maps the weights into its own address space, but there is only one physical copy in HBM. GMS holds the allocation and hands out the handles; it does not sit between an engine and the memory it reads from

This architecture provides two benefits. First, weights persist beyond engine failure. While the kernel removes the failed engine’s CUDA context, the GMS reference ensures the physical pages stay resident so a fresh engine can immediately map them. Second, weights can be shared between concurrent engines ; therefore, a secondary engine on the same GPU incurs zero marginal weight cost.

Integrating GMS into inference frameworks requires only a narrow change. vLLM, SGLang, and NVIDIA TensorRT-LLM each integrate GMS through a custom torch.cuda.CUDAPluggableAllocator bound to the weight memory pool. From inside the engine, weights remain ordinary torch.Tensor s. Adopting GMS is as simple as flipping a flag at startup.

GMS is not limited to weights. The current preview does not support using GMS for the KV cache, but that capability is under active development. The goal is for a promoted shadow to map the outgoing engine’s cache instead of rebuilding it as traffic arrives.

A shadow engine is a fully initialized engine process that remains idle and co-resident on the same GPUs as the active engine. Weight sharing makes this configuration feasible: without it, the second engine would require another full copy of the weights, substantially reducing the memory available for processing requests.

A shadow runs through the same startup path as an active engine. On each of its GPUs, it connects to the local GMS and imports the weight mappings, establishes communicators (NCCL and NIXL for KV transfer between workers), captures CUDA graphs, and performs any necessary warm-up. By the end of the startup, it is ready to serve. Then, instead of serving, it parks: it releases the materializable parts of its memory and blocks, waiting for its turn.

What a shadow has precomputed before it parks:

What it has deferred:

A parked shadow therefore retains only its CUDA context, captured graphs, communicators, and weight mappings—no separate copy of the weights and no KV cache. This footprint is small enough for the shadow to remain alongside an active engine on the same devices, enabling recovery within seconds.

The following figure shows how these components fit within each worker and scale behind a shared router.

Figure 3. A fleet of workers behind a single router. The two-engine layout is internal to each worker, so the router, frontend, and orchestrator need no changes to benefit from it

These foundations are integrated into one pod. The worker holds two engine containers, a GMS sidecar to mediate GPU memory access, and a shared lock to elect the active engine.

At steady state, one engine holds the lock and remains awake, connected to GMS, holding a materialized KV cache, and registered with the frontend router. The other remains fully initialized and connected to GMS but is dormant, holds no KV cache, and waits on the lock.

The following sections trace the recovery sequence and explain the synchronization and memory-management mechanisms that make it reliable.

A worker moves through four phases before returning to steady state.

Figure 4. The four phases of a recovery. Active and shadow roles swap between engine A and engine B, and the worker returns to steady state without either engine reloading weights

T₀ Steady. Engine A holds the lock and is awake, registered with the router. Engine B is dormant, blocked on the lock.

T₁ Failure. Engine A’s process exits, either because it crashed outright or because a liveness probe found it hung and killed it. Either way, the kernel releases its lock as the process is reaped. The worker is briefly unroutable, until the shadow registers.

T₂ Cutover. Engine B acquires the lock, wakes, remaps weights through GMS, materializes its KV cache, and re-registers with the router. Engine A’s container is restarted by the orchestrator.

T₃ Restarted. Engine A finishes initialization and enters the shadow state. The system returns to a steady state, with the roles swapped.

The shadow’s advantage is that it enters T₂ already initialized. The only work on the critical path is acquiring the lock, remapping weights, and materializing the KV cache.

The worker requires both mutual exclusion, ensuring only one engine is awake at a time, and reliable release to ensure the standby engine takes over if the active one fails. A POSIX flock on a shared file provides these guarantees. When the active process exits due to a shutdown, segfault, or SIGKILL , the kernel reaps its file descriptors, and the shadow engine acquires the lock to begin serving.

Each engine’s startup path is therefore a short leader election:

await engine.initialize() # weight load, torch.compile, autotune, CUDA graph capture ... # put the engine to sleep while we wait on the lock await engine.sleep() lock = FlockFailoverLock(lock_path) await lock.acquire(engine_id=engine.id) # wait on the lock to wake await engine.wake()

A deadlocked engine whose process is still alive falls to the Kubernetes liveness probe, which cascades to a SIGKILL and trips the same kernel-managed release.

Fitting two engine processes on one GPU without exhausting HBM takes careful accounting across the lifecycle.

Figure 5. GPU memory across a recovery

Weights. Allocated once by GMS and mapped read-only by every engine in the worker; never duplicated.

KV cache. Held only by the active engine today: materialized when it wakes, released when it dies, freeing the region for the shadow to take over.

Buffers and graphs. NCCL buffers, the CUDA context, and captured graphs. Held by each engine even while dormant, and the whole of a parked shadow’s standing cost.

Benchmark results: shadow engine recovery vs. cold restart on GLM-5.2

To quantify the benefit, we compared shadow engine recovery with a cold restart after an engine failure in a two-worker fleet.

We ran two workers serving GLM-5.2 quantized to NVFP4 on NVIDIA B200 nodes: one worker per node, TP=8, 200K max context, and an FP8 KV cache. A single frontend distributes requests round-robin across the two. The load is synthetic: 32,000 input tokens and 1,000 output tokens per request, arriving at 0.7 requests per second.

Both arms run identical engine builds and configurations; the only difference is the shadow engine. The baseline has it off, and the killed worker cold-restarts. In the Shadow Engine Recovery configuration, each worker pod hosts a preinitialized shadow engine that can take over if the active engine fails.

We injected the fault only after the workload reached its steady-state operating point: a SIGKILL to one of the two workers, followed by 600 seconds of observation. With two workers in the fleet, losing one leaves the survivor carrying every request until its partner returns.

Figure 6. Time to first token, 60-second rolling median. The baseline climbs for the whole 283 seconds its second worker is missing, crossing the 5-second line about 90 seconds in. With shadow engine recovery, p50 TTFT is more resilient to disruption

Figure 7. Inter-token latency, 60-second rolling median. The baseline steps to 84 milliseconds and holds flat for 283 seconds, well above the 50-millisecond line. The shadow arm rises briefly at the fault and settles back

Metric Baseline: cold restart Shadow engine recovery Time until a second worker serves again 283 s 7.3 s TTFT p50, after the fault 23,815 ms 1,311 ms Decode rate p50, after the fault 12 tok/s/user 46 tok/s/user Requests over 5s to first token 201 of 399 1 of 398 Requests below 20 tok/s/user 226 of 399 0 of 398 Table 1. Cold restart versus shadow engine recovery in the window after one of two workers is killed

Compared to the cold restart baseline of 283 seconds, shadow engine recovery only takes 7.3 seconds (1.7 seconds to detect the fault and 5.6 seconds to promote the shadow). This results in significantly better TTFT and decode rate after the fault, and allows us to largely avoid SLA violations.

Having validated that fast recovery for inference workloads on Kubernetes is feasible, we are working to stabilize the implementation and expand support to a wider range of workloads. Shadow engine recovery will roll out incrementally over the coming months.

The current preview has several limitations and deployment requirements.

To try Shadow Engine Recovery, start with the Kubernetes quickstart to create a running deployment. Then follow the Shadow Engine Recovery deployment workflow and use the vLLM failover example as a complete manifest. Visit the ai-dynamo/dynamo repository to ask questions, report issues, or contribute.