NVIDIA 发布 NeMo Switchyard，为 AI 智能体提供跨模型路由

Technical Blog

sessionStorage.removeItem('facets.search.text'); sessionStorage.removeItem('facets.search.text-copy'); sessionStorage.removeItem('facets.search.filters');

Learn how NVIDIA NeMo Switchyard routes agent workloads across specialized and frontier models to balance performance, cost, and efficiency.

AI-Generated Summary

Powered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. Learn more

Building an AI agent does not end with choosing a single model. Each model has its own strengths, weaknesses, and cost profile, which can shift from one workload to another—or even within the same workload. For example, an agentic task may need classification for one step, reasoning for the next, and a smaller model for routine follow-up tasks. Sending every request to the largest model can increase cost and latency, while sending every request to a smaller model can reduce quality on complex tasks.

Model routing addresses this challenge by orchestrating specialized and frontier models so that each task uses the model best suited to each task. NVIDIA NeMo Switchyard makes this complex engineering problem practical for agent workloads, so developers can route work across models without rebuilding their applications around each provider or model choice.

At runtime, a router evaluates each request and its available context, then sends the work to the model that best suits the task’s requirements, constraints, and policies. Depending on the workload, this system of models may improve accuracy and reduce cost compared with using the most capable model for every request.

NeMo Switchyard provides a library for applying multiple routing approaches. This post explores how NeMo Switchyard enables developers to apply a system-of-models approach and build more efficient, controllable agents better suited for real AI workflows.

Consider a system of models performing a computer-use task measured by the Terminal-Bench Hard benchmark (Figure 1). While DeepSeek V4 has the highest overall accuracy in this example, it isn’t the best model for every task group. For instance, Kimi K2.6 is better suited to the ML and RL task groups, while Qwen3.5 397B A17B is preferable for math and science. The remaining six task groups should use DeepSeek V4. This approach can also be applied at the individual-task level or across phases of solving a single task.

Figure 1. Model accuracies on different task groups on the Terminal-Bench Hard with the Terminus agent

Cost and completion time further complicate the decision, as shown in Figure 2. Each model has its own costs associated with running or accessing the model, as well as its own verbosity profile, relating not just to tokens but also tool calls.

Figure 2. Model accuracy vs total prompt tokens (left) and total output tokens (right) on Terminal-Bench Hard with the Terminus agent (STD bars are also shown)

Building a router requires considering signals from various sources. Each routing algorithm helps determine where to derive those signals.

Broadly, effective routing decisions rely on signals from three areas:

To understand the capability and cost signals:

Importantly, a router must consider not only which signals to use, but also when and where to evaluate them. For instance, for a multi-turn agent task, the router may route each full request to a specific model or route at each step. The whole system may share a pool, or sub-agents may use specialized model pools for the tasks.

The answers to these questions depend on several factors, including the use case, deployment complexity, error tolerance, latency, or throughput constraints. The system also requires infrastructure for a seamless and invisible handoff to the agent/user.

NeMo Switchyard solves these challenges with an intelligent orchestration layer that supports multiple routers. Developers can also bring their own routing algorithms or customization data to NeMo Switchyard.

Routing algorithms produce signals that inform routing decisions across models with different strengths. The system also needs infrastructure that can take a router’s answer, send the request to the chosen model, and carry the response back to the application.

This starts with NeMo switchyard-libsy, the provider-agnostic SDK behind NeMo Switchyard. It represents requests, defines the models available to a system, and manages calls to the selected model. Each model target has a semantic name, while the client behind it maps that name to the provider endpoint and model ID. This separation keeps the routing logic independent of a specific provider.

Figure 3. Architecture diagram for NeMo Switchyard

NeMo Switchyard can carry routing state across an agent’s session when a policy requires it. It can also retain information from earlier turns, such as tool results or an affinity decision, and make that context available for later routing decisions. A route can remain stateless when that history isn’t needed.

This separation is important because model deployments change. A team may update a model, move it to another endpoint, or use a different provider without changing the routing integration. NeMo Switchyard can make a model call through the target client or return the call to the host application. This gives an agent runtime, inference platform, or gateway control over how requests are served while retaining the same routing contract.

The NeMo Switchyard server is a reference for making routing available through common APIs, to simulate an LLM gateway. It accepts OpenAI, Anthropic, and Responses API requests, translates them into the internal NeMo Switchyard request format, and returns the expected response format. It also records the selected model, decision rationale, token usage, latency, and call outcomes, so teams can inspect a running route.

With the infrastructure in place, the next step is looking at the routing approaches that use it to make decisions. NeMo Switchyard offers both tuning-free and tunable routers.

NeMo Switchyard includes several tuning-free routers that make decisions without training on workload-specific data, including the LLM classifier, stage router, and escalation router.

The LLM classifier uses an LLM as a judge to select a candidate LLM and maintains session affinity with that model across later turns. This avoids repeatedly reclassifying work that has not materially changed throughout the arc of an agent solving a task.

This approach fits headless and domain-specific systems. A team can route coding, mathematical, or healthcare work tasks to selected model targets, while NeMo Switchyard supplies the routing and state-management pieces.

A coding agent moves through different stages. Early on, it explores the codebase and recovers from errors. Later it settles into a more mechanical implementation. These stages require different levels of model capability, which the stage router uses to make routing decisions.

For each turn, the stage router examines recent tool activity to decide how much model capability the agent needs. Severe errors, repeated unproductive work, or prolonged exploration push the turn toward the capable model. Steady writes and edits, especially once tests are passed, favor the efficient model. If the signals are inconclusive, the router can consult an LLM judge before falling back to its configured default.

Escalation routing starts each conversation with a lower-cost model. An LLM judge monitors the progress of the task, turn by turn, and moves the session to a more capable model when it detects sustained difficulty.

This approach is designed for multi-turn agent workloads in which a smaller model can handle routine work but may need support after repeated errors, loops, or drift, extending the LLM classifier routing approach from static to adaptive.

Figure 4. NeMo Switchyard performance benchmark

Tunable routers

Tunable routers build on this routing foundation by replacing fixed heuristics with signals learned from real-world workload data. Rather than determining which model is most appropriate based on the request text, a tunable router can learn to predict how likely each candidate model is to correctly answer a request.

During training, the prefill router extracts the LLM’s residual stream to estimate the complexity of the query. A shared-trunk MLP uses signals from the residual stream and maps them to accuracy labels for each LLM in the routing pool.

At inference time, the prefill states act as ‌input to the router, and the shared trunk predicts the likelihood that each LLM will successfully complete the task or answer the query.

The router can then apply a policy that blends predicted accuracy with cost, latency, or other deployment constraints. Each candidate model receives a score, and the request is routed to the model with the best tradeoff for that workload. Figure 5 shows that routing is not just about picking the strongest model. It is about choosing the model most likely to meet the required quality level at the right cost.

Figure 5. Accuracy versus total test-set cost for a learned prefill router on personal assistant-type tasks (Pinchbench + ClawdQA) https://arxiv.org/abs/2603.20895

Improving agent efficiency with NeMo Switchyard

NVIDIA is working with partners across the agent, model, and enterprise application ecosystem to bring NeMo Switchyard model routing into existing developer workflows without a separate setup. These collaborations include:

LangChain benchmarked NeMo Switchyard using its internal deep agents evaluation suite, which includes 145 multi-turn agentic tasks that reflect production workloads, such as customer support dialogue under policy constraints, on-call incident investigation, and multi-step workflow automation across messaging, issue tracking, and email. The suite evaluates tool use, multi-step retrieval, filesystem operations, and long-context summarization, with scenarios drawn from τ²-bench airline, Berkeley Function Calling Leaderboard, FRAMES, and Nexus. Across five runs, routing requests between NVIDIA Nemotron 3.5 Lightning and Claude Opus 4.8 with the escalation router delivered a 74% cost reduction compared with a frontier-only baseline across five runs, sending just 7% of calls to the frontier model, at a measured ~6-point accuracy tradeoff.

Cognition implemented the NeMo Switchyard staged-routing methodology in Devin Desktop and deployed it to NVIDIA internal users for real-world testing. On FrontierCode Main, Cognition’s benchmark for production-grade coding tasks, the implementation routed between Opus 5 and Kimi K2.7. It delivered near-frontier performance, achieving 50.6% at a $3.11 mean cost—within 2.8 percentage points of Opus 5 accuracy at approximately 28% lower mean cost. Together, the benchmark and internal deployment provide a practical case study in model-neutral and adaptive agents, showing how the complementary strengths of different models can be applied dynamically as a task evolves.

Developers can start with partner integrations that bring NeMo Switchyard into familiar agent tools, frameworks, and gateways, or build custom routing into their own agents and LLM gateways using the NeMo Switchyard GitHub instructions .

Model routing enables systems of specialized and frontier models to work together, delivering results that are greater than the sum of their parts. However, building a useful and production-ready routing system remains a very difficult engineering challenge.

As AI systems increasingly combine models, routing is essential for selecting the right model for each task while balancing efficiency, quality, and cost.