Data Center / Cloud How NVIDIA Uses AI Agents to Streamline AI Factory Operations Nov 04, 2026 By Kavya G Y and Rohan Adwankar Discuss (0)

AI-Generated Summary

  • The AI Factory Operations Agent coordinates specialized sub-agents to detect failures, correlate root causes across compute, network, hardware, and scheduling layers, and surface recommendations through a chat interface for NVIDIA DGX SuperPOD operations.
  • The framework is built on the NVIDIA NemoClaw blueprint using NVIDIA OpenShell and runs fully on-premises with locally served NVIDIA Nemotron open models across NVIDIA DGX H100, NVIDIA DGX B200, and NVIDIA DGX GB300 production clusters.
  • Domain-specific agents include Job Recovery, Hardware, Observability, Research, Kubernetes, and Cluster Management agents, each providing targeted recommendations while integrating cross-cutting MCP tools such as web search, code execution, and Grafana dashboards.
  • Production learnings emphasize clear agent boundaries, read-only investigation to build trust, cross-layer correlation for root causes missed by isolated tools, precise MCP tool definitions, and transparent reasoning so SREs can review evidence before acting.
  • For multi-tenant AI factories, scoped agents, tools, sandboxes, and Kubernetes permissions isolate tenant data and operations while central operations retain broader cluster visibility, enforced by OpenShell kernel-level sandbox restrictions.

Next Steps

Powered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. Learn more

As GPU clusters scale, operations span compute, network, hardware, and scheduling. Site reliability engineers (SREs) often have to examine multiple dashboards to identify the cause of training performance drops. What if a team of AI agents could perform that correlation through one interface?

NVIDIA developed the AI Factory Operations Agent to streamline its own internal NVIDIA DGX SuperPOD operations and reduce the manual effort by NVIDIA SREs. Sub-agents communicate through Model Context Protocol (MCP) that any organization can adopt to maximize AI factory uptime and performance.

Instead of requiring SREs to check each layer of the AI factory software and hardware separately, the AI Factory Operations Agent uses specialized agents for different operational domains. In production, these agents detect failures, correlate root causes across layers, and surface clear recommendations through a chat interface. SREs can interact with any agent using natural language, switch between investigations, and track every action for auditability.

This blog will share learnings and teach you how to implement the AI Factory Operations Agent in your own workflow. 

The architecture: Using NVIDIA blueprints and open models

The AI Factory Operations Agent developer example was built around the NVIDIA NemoClaw blueprint. At the center sits the user-facing agent runtime, which receives SRE queries through a chat interface and invokes the appropriate domain agents and tools.

Agents connect to infrastructure through OpenClaw plugins and MCP servers, with each MCP exposing a focused set of approved operational tools. You can work through the built-in web UI or integrate existing coding-agent interfaces through the API. This modular design lets you add new agents and infrastructure integrations without changing the existing agents.

The framework is built using NVIDIA OpenShell and is fully on-premises capable, supporting extensibility and data sovereignty. Agent inference uses locally served NVIDIA Nemotron open models, while the framework runs across internal NVIDIA DGX H100, NVIDIA DGX B200, and NVIDIA DGX GB300 production clusters. See the next section for how OpenShell provides agent governance and controls so agents work safely.

Diagram of workflow where either an SRE or agent triggers the AI factory agents powered by NVIDIA Nemotron 3 models that perform reasoning tasks to output root cause analyses, run commands for issue resolution, or generate relevant dashboards.
Figure 1. AI Factory Operations Agent architecture: A NemoClaw agent orchestrating domain-specific agents across GPU cluster infrastructure

How NVIDIA built it

The AI Factory Operations Agent runs as a Kubernetes control-plane application. Helm deploys the UI, OpenClaw gateway, MCP tool servers, inference endpoint, and NVIDIA OpenShell runtime as Kubernetes services. The chat UI sends each request to NemoClaw, which routes it to the appropriate domain agent. 

For shell execution, OpenShell creates a per-session Kubernetes sandbox instead of running commands in the main gateway pod. The runtime receives the AI Factory Operations Agent workspace, selected skills, and command policy. Its lifecycle does not modify existing services or production workloads.

Security is enforced in layers. Kubernetes inspection uses the server-side run_kubectl tool, which invokes a pinned kubectl binary against registered read-only credentials. OpenShell receives neither kubectl nor Kubernetes credentials. Kubernetes service accounts and role-based access control (RBAC) define the cluster resources the tool can inspect.

Because these checks run through the same Kubernetes access path used by the runtime, the results show what the agent can and cannot do inside the cluster.

(agent) $ kubectl auth can-i get pods
yes
(agent) $ kubectl auth can-i get configmaps
yes
(agent) $ kubectl auth can-i get secrets
no
(agent) $ kubectl auth can-i create pods
no
(agent) $ kubectl auth can-i update pods
no

The AI Factory Operations Agent, sub-agents, and tools

Internal NVIDIA deployment of the AI Factory Operations Agent uses a growing ecosystem of NVIDIA data center domain-specific agents, each providing recommendations.

Job Recovery Agent: Detects training job failures and goodput degradation; performs multi-layer root cause analysis across scheduler, application, hardware, and network layers; and recommends recovery steps when root cause is confirmed.

Hardware Agent: Triages hardware faults at the device level—PCIe errors, ECC errors, GPU issues, NVMe failures, and thermal anomalies—tracing symptoms down to the specific component.

Observability Agent: Queries historical Prometheus, telemetry, and integrated Grafana Loki logs to create dashboards dynamically and display them directly in the web UI.

Research Agent: Searches preloaded documentation including NVIDIA Mission Control manuals and DGX SuperPOD reference architectures and deployment guides. You can also upload your own runbooks and site-specific guides.

Kubernetes Agent: Provides read-only operational awareness across pods, services, deployments, GPU resources, node status, and scheduling state.

Cluster Management Agent: Interfaces with NVIDIA Base Command Manager APIs through diagnostic MCP tools for advisory cluster management and troubleshooting.

Beyond the domain agents, the framework integrates cross-cutting MCP tools including web search, code execution, and Grafana dashboards.

You can also connect your own MCP servers and APIs, adapting the framework to the schedulers, observability platforms, cluster managers, and operational procedures already used in those environments.

Learnings from using agents in the data center

NVIDIA continues to refine the framework by measuring root cause analysis (RCA) accuracy, tracking SRE agreement, and expanding the agent ecosystem. From production use, five lessons stand out:

  • Clear agent boundaries: Overlapping ownership creates noise, not signal.
  • Read-only first: Accurate investigation builds SRE trust before automation.
  • Cross-layer correlation: The framework connects workload, hardware, and network signals to find root causes missed by isolated tools.
  • Precise MCP tools: Clear tool definitions improve routing and reduce wrong tool selection.
  • Transparent reasoning: SREs need to see tool calls, evidence, and conclusions before taking action.
Video 1. AI Factory Operations Agent incident trouble-shooting example, injected into an NVIDIA DGX cluster for internal demonstration.

Bringing agent-driven operations to your AI factory

For organizations running AI factories at scale, the preferred deployment model is a dedicated GPU-accelerated control plane isolated from the production cluster. The control plane handles agent orchestration and reasoning while the production cluster provides telemetry without consuming training GPUs or disrupting workloads. 

Internal NVIDIA deployments run model inference on dedicated GPU capacity and use the existing control-plane CPU Kubernetes admin space for runtime services. Other deployments can use a local or cloud inference endpoint based on data sovereignty, cluster size, and available GPU capacity, with smaller deployments using an RTX PRO Workstation or DGX Spark.

For multi-tenant AI factories, scoped agents, tools, sandboxes, and Kubernetes permissions control the cluster data, tools, and nodes available to each tenant. Tenant SREs see only their namespaces, jobs, and nodes, while central operations retain broader cluster visibility. Each tenant can use its own set of domain-specific agents and dedicated inference hardware when required. OpenShell enforces these boundaries by running each tenant’s agents and tools in an isolated sandbox with kernel-level restrictions.

Get started

To get started on streamlining cluster operations management, explore the AI Factory Operations Agent developer example, or access the open source framework and code directly on GitHub. The orchestration framework is available as experimental open source software; APIs, configuration, and deployment requirements may change between releases. NVIDIA customers can contact their account manager to request access to NVIDIA-developed cluster operations agents.

Discuss (0)

Tags

Agentic AI / Generative AI | Data Center / Cloud | MLOps | HPC / Scientific Computing | Nemotron | Intermediate Technical | Tutorial | AI Agent | AI Factory | Build AI Agents | DGX | GB200 | GB300 | HGX | Machine Learning & Artificial Intelligence | NemoClaw | OpenShell

About the Authors

Avatar photo About Kavya G Y
Kavya G Y is a Product Manager for NVIDIA enterprise products, focused on AI infrastructure, next-generation GPU platforms, and technologies that enable customers to deploy and operate AI at scale. Her work spans the AI stack, from hardware to models and applications, and brings together the technologies needed to build and operate enterprise AI factories. She is particularly interested in AI inference and serving, agentic AI, physical AI, and emerging technologies that make enterprise AI more scalable, efficient, and easier to operate. Kavya holds a master’s degree in computer science from the University of Southern California. View all posts by Kavya G Y Avatar photo About Rohan Adwankar
Rohan Adwankar works on agentic systems and observability for AI infrastructure at NVIDIA. He is interested in scaling inference and training workloads across hardware and software and how agents can help with these goals. He graduated from the University of California, Los Angeles. View all posts by Rohan Adwankar

Comments