NVIDIA previews 7.3-second LLM engine failover

NVIDIA has published details of shadow engine recovery, a preview feature in its Dynamo inference framework that keeps a fully initialized standby engine parked on the same GPUs as the active one, ready to take over when the serving process dies.

The mechanism rests on a per-GPU sidecar called the GPU Memory Service (GMS), which owns physical GPU memory on behalf of engines using the CUDA Virtual Memory Management API. Because physical allocations are reference-counted and their lifetimes decoupled from any single CUDA context, weights stay resident in HBM when an engine process exits, and two co-resident engines can map the same weight tensor without a second copy. Non-transferable state — CUDA context, captured graphs, NCCL and NIXL communicators — is built by the shadow before the failure, then it releases materializable memory and blocks.

In NVIDIA's test, one worker in a two-worker GLM-5.2 deployment was deliberately terminated. A cold restart took 283 seconds, during which the surviving worker absorbed all traffic, raising TTFT and cutting per-user decode rate. With shadow engine recovery, a second worker resumed serving in 7.3 seconds — roughly 39x faster.

Integration is described as narrow: vLLM, SGLang and TensorRT-LLM bind GMS through a custom torch.cuda.CUDAPluggableAllocator on the weight memory pool, with weights still appearing as ordinary torch.Tensors inside the engine.

The preview does not yet cover the KV cache, which NVIDIA says is under active development; the stated goal is for a promoted shadow to map the outgoing engine's cache rather than rebuild it. No general-availability timing was given.