NVIDIA 如何用 CLI、Skills 和 AI 编码智能体开发 Holoscan 应用

Technical Blog

sessionStorage.removeItem('facets.search.text'); sessionStorage.removeItem('facets.search.text-copy'); sessionStorage.removeItem('facets.search.filters');

Aug 19, 2026

HoloHub reference applications and the Holoscan CLI enable an engineer-guided, agentic development workflow for real-time AI applications at the edge. An endoscopic tool segmentation dashboard was built in three iterations: a minimal working application reusing the MONAI model and sample video, a benchmark mode with repeatable latency measurements, and latency optimizations that improved rendered throughput by 50.5% and reduced mean application-path latency by 33.6%. The workflow uses ./holohub commands for scaffold creation, execution modes, testing, and environment capture, allowing the engineer to review the same artifacts the agent produces. An ablation study showed that combining the CLI, HoloHub development skills, and documentation yielded the fastest agentic processing time (40 minutes) and lowest token usage (11M) compared to configurations without skills or CLI guidance. Next Steps Explore the HoloHub repository for reference applications and components. Review the HoloHub development skills to guide agentic development workflows. Try the Holoscan CLI to create, run, and test HoloHub applications.

Powered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. Learn more

NVIDIA Holoscan is a platform for building real-time AI applications at the edge, from medical imaging to robotics. HoloHub is its companion repository: a growing collection of reference applications and components that demonstrate what’s possible.

We wanted to explore how a general-purpose coding agent could use the same examples, documentation, and development tools available to an engineer in an actual development task.

In this post, we walk through building a real-time endoscopic tool segmentation application using an AI coding agent. HoloHub examples and documentation provide implementation patterns, and the development skills guide the agent through the HoloHub development process.

The Holoscan CLI , invoked via ./holohub wrapper, provides the shared execution interface. The agent can discover development operations through the CLI, while the engineer can inspect and repeat the same commands.

The development workflow proceeds in iterations:

The workflow is agent-agnostic; in this example, we used Codex with GPT-5.6 sol max mode, and the agent processing times mentioned were approximate.

Figure 1. AI-assisted development and review workflow

Setup and development target

The overall objective is an end-to-end endoscopic tool segmentation application: real-time inference with a live visualization of the segmentation masks, along with statistical analysis rendering.

We reused the existing MONAI endoscopic tool segmentation model and a Holoscan sample video, and confirmed that the existing application monai_endoscopic_tool_seg worked locally. We focused on a new application reusing the deep learning segmentation pipeline, and adding comprehensive visualization, runtime telemetry and repeatable benchmarking.

The agents were additionally provided with:

The next sections show how the pieces work together in an engineer-guided, agentic development workflow.

Instead of trying to build the entire application with a single prompt, developers should decompose the ultimate objective into smaller, verifiable engineering iterations guided by uncertainty and evidence. This approach ensures that design choices are reviewed in a timely manner.

The comprehensive goal can thus be structured into a sequence of reviewable topics:

Each iteration produces reviewable code, outputs, and tests, which inform the prompts and design choices for the next iteration.

The first prompt defined the outcome while constraining model and data reuse.

Developer prompt 1:

Use $holohub-app-lifecycle to create a separate new Python HoloHub application for displaying endoscopic tool tracking as model outputs https://github.com/Project-MONAI/model-zoo/tree/dev/models/endoscopic_tool_segmentation. Reuse the MONAI endoscopic tool segmentation model, sample data, preprocessing, and inference. Show the model-derived mask, coverage and timeline, and useful uncertainty measurements in a polished HoloViz overlay. Do not train or modify the model weights. Make the sample video work end to end.

That left implementation choices to the agent while keeping model reuse, visual evidence, and weight integrity explicit.

The agents collected information from the designated sources: read the app lifecycle skill, nearby HoloHub examples, project metadata, and CLI documentation.

Different types of actions were taken as expected:

The resulting application connected video replay, preprocessing, TensorRT inference, the SDK segmentation postprocessor, telemetry, and HoloViz. Inference and mask postprocessing ran for every replayed frame. The overlay reported frame-derived measurements.

Figure 2. Endoscopic tool segmentation dashboard

The agentic processing time was 40 minutes. The developer could review the live app through the same CLI used by the agent:

./holohub run endoscopy_tool_segmentation_dashboard visual --language python

After reviewing the implementation, the visual outputs, and the test cases, we confirmed that the reused model and sample video worked in the new application. Visual review also showed that the overlay needed clearer measurements, so we proceed to define the next iteration.

The second prompt turned the visual demonstration into a repeatable development artifact:

Developer prompt 2:

Revise the visual output, add more meaningful statistics, tool area, mask motion, temporal intersection-over-union as a stability indicator, edge entropy, FPS, bounding box position, and remove values that remain unchanged during replay. Add a benchmark mode that records actual latency and plots the results in Python. Export the figures to the build folder and also show them interactively when the environment supports it.

In response, the agent revised the dynamic measurements and screenshot readability, then made visual review and benchmarking explicit application modes. The app was implemented to have three named application modes:

With the Holoscan CLI and application lifecycle management, the modes and tests are discoverable and runnable without remembering a detailed container and application script recipe:

./holohub modes endoscopy_tool_segmentation_dashboard --language python ./holohub run endoscopy_tool_segmentation_dashboard benchmark --language python ./holohub test endoscopy_tool_segmentation_dashboard --language python

Benchmark mode used Holoscan Data Flow Tracking for the configured path from the video replayer through preprocessing, inference, telemetry, offscreen HoloViz, and the rendered-frame sink. It effectively reused the ideas presented in the existing holoscan flow benchmarking module . The agentic processing time was 20 minutes.

Figure 3. Performance benchmark of the endoscopic application, comparing application-path latency over 300 tracked frames

The revised benchmark provided more repeatable measurements. With that baseline available, the next iteration could investigate performance without relying on visual checks alone.

Once the application was measurable, the developer issued a third prompt:

Developer prompt 3:

Check whether the deep-learning model runs on every frame. Investigate ways to reduce latency, including approaches that take advantage of similar neighboring segmentations, and show the new benchmark results.

In response, the agent confirmed that inference still ran on every frame. It considered reusing masks across neighboring frames, which could avoid some inference work but would require a policy for deciding when stale output is acceptable. For this engineering iteration, it kept inference on every frame and first removed lower-risk dashboard overhead:

We compared the first iteration’s implementation with the final one on the same test system. The optimized version was faster in all five measured trials. The agentic processing time was 30 minutes.

Figure 4. Final performance comparison between the baseline and optimized versions of the endoscopy dashboard

Final handoff prompt and revalidation

After the three engineering iterations, the developer gave the agent a separate handoff prompt.

Developer prompt 4:

Commit the implementation and keep the benchmark logs.

To finalize the progress, we reran the application and the tests, verified the output figures and headless test results. The implementation and benchmark evidence were retained along with the commit hash and dependency versions via ./holohub env-check and ./holohub env-info .

The development iterations show what the workflow produces. To better understand the impact of the CLI, skills and documentation, we next compare the same development task under different combinations of these resources.

We compared resource usage using the same coding agent and sandbox environment across identical settings (note that the study was last conducted on Aug. 1, using Codex 0.146.0 with GPT-5.6 Sol at max reasoning effort). All evaluations were based on the single prompt used in the first iteration (with the skill name removed if unavailable) with the goal of creating a new application.

The agentic processing time was 40 minutes, with a total cost of 11M tokens.

The agents were given agents.md including references to CLI usage guide and docs, as well as the HoloHub codebase. No HoloHub development skills were provided.

The agentic processing time was 65 minutes, with a total cost of 20M tokens. While the workflow resulted in a complete application that achieved the implementation goal, the process was less efficient. The agents correctly located similar apps within the HoloHub repository, but tended to use generic Bash tools that required more trial-and-error probing of the development environment. For instance, they often applied generic linting tools before CLI-based linting, or attempted to install Python dependencies and run inference scripts directly on the host (should have consistently worked in a container).

The agents were given the HoloHub codebase but without the agents.md and explicit guidance on the CLI usage. No HoloHub development skills were provided.

The agentic processing time was 40min, with a total cost of 15M tokens.

No CLI guidance was provided, the coding agents still gained understanding of the CLI (from the overall codebase examples) and used it as the main dev tooling. However the coding quality is suboptimal compared to the other two settings:

While follow-up prompts could potentially address these issues, the combination of CLI, skills, and documentation/examples provided the best developer experience with the lowest resource overhead.

This post follows a small, verifiable development loop. The result is an engineering prototype built on an existing model, sample video, and Holoscan components. It runs end to end, provides several application modes and automated tests, preserves the model weights, and records reproducible benchmark evidence.

The main takeaway is the development loop shared by the developer and the agents: ./holohub supplies consistent operations, the skill encodes project-specific sequences and checks, and examples and documentation provide engineering context. The agent and the developer use the same CLI commands. The developer can focus on defining the objectives, setting the constraints, evaluating trade-offs, and deciding whether the evidence is sufficient.