Understand this story
On November 3, 2026, NVIDIA released the open-source workload-level recovery service NVIDIA Recovery. The service detects training interruptions through application logs, GPU metrics, and scheduler signals, and processes them in a four-phase loop: detection, attribution, isolation, and restart or stop. The service is released under the Apache 2.0 license in the dsx-ai-factory/NVIDIA-Recovery repository; Slurm integration is implemented through an Advisor Controller deployed via a Helm chart, supporting both in-cluster and SSH modes. New deployments default to monitoring mode, which only outputs recovery recommendations and does not execute scheduling actions. From July 9 to October 9, monitoring on a training cluster covered nearly 200 jobs per day, recording approximately 8,550 anomaly detections, with average detection and recovery action latencies of 69 seconds and 94 seconds, respectively. The service is currently released as open source, and there is no information on further developments yet.
AI-generated from public reports
Latest development · Report
NVIDIA Open-Sources NVIDIA Recovery to Automatically Recover Distributed Training Jobs on ClustersDevelopments
1 confirmed developments
NVIDIA open-sources and releases the NVIDIA Recovery workload recovery service
Source:NVIDIA Developer · Generative AI
Reports and discussion timeline
Browse reports and discussion by time. Their evidence levels remain distinct.
Nov 3
NVIDIA Developer · Generative AIFirst-partySelected
NVIDIA Open-Sources NVIDIA Recovery to Automatically Recover Distributed Training Jobs on Clusters
NVIDIA has released NVIDIA Recovery, an open-source workload-level recovery service that detects training interruptions through application logs, GPU metrics, and scheduler signals, and handles them in a four-stage loop: detection, attribution, isolation, and restart or stop. The service is available under the Apache 2.0 license in the dsx-ai-factory/NVIDIA-Recovery repository. Slurm integration is implemented through an Advisor Controller deployed via a Helm chart, supporting both in-cluster and SSH modes. New deployments default to monitoring mode, which only outputs recovery recommendations without executing scheduling actions; from July 9 to October 9, monitoring on one training cluster covered nearly 200 jobs per day and recorded approximately 8,550 anomaly detections, with average detection and recovery action latencies of 69 seconds and 94 seconds, respectively.