NVIDIA 发布 Isaac for Healthcare 医疗物理仿真框架

Technical Blog

sessionStorage.removeItem('facets.search.text'); sessionStorage.removeItem('facets.search.text-copy'); sessionStorage.removeItem('facets.search.filters');

Related Resources

Simulation / Modeling / Design

English 中文

Jul 28, 2026

Endoluminal Simulation Module enables real-time simulation of flexible catheter navigation through patient-specific vasculature using GPU-accelerated XPBD solvers. Surgical Simulation Module models soft-tissue deformation and surgical interactions with haptic feedback at over 30 frames per second on a single consumer GPU. Cosmos-H family of generative world models produces realistic surgical video from robot actions, with Cosmos-H-Dreams enabling interactive simulation at more than 30 frames per second via FlashDreams inference engine. Classical physics solvers and generative simulation complement each other, providing deterministic physical guarantees and diverse visual realism respectively for healthcare robotics training. Unified simulation loops combine anatomical digital twins, device mechanics, and fluoroscopic imaging in a single GPU-resident pipeline for high-throughput reinforcement learning. Next Steps Build an endoluminal robotics workflow using the catheter navigation workflow to create patient-specific simulations. Run Cosmos-H Dreams interactive simulation from the GitHub repository with pretrained checkpoints. Download Cosmos-H Dreams model checkpoints from the Hugging Face checkpoint for generative surgical simulation.

Powered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. Learn more

Unlike autonomous driving or industrial robotics, healthcare robotics can’t rely on internet-scale data collection or unlimited real-world experimentation. Every demonstration requires specialized equipment, clinical expertise, and access to patients or laboratory environments. This creates three fundamental challenges for developers.

First is the data gap . Training modern robotic policies requires demonstrations across diverse anatomies and procedures. Most teams have only hundreds of demonstrations—not the tens of thousands needed to build robust systems. But even if collecting millions of demonstrations were practical, the most important cases would still be missing. This is because healthcare is dominated by the long tail . Rare anatomies, challenging patient physiology, complications, and failure modes occur too infrequently to be adequately represented in real-world datasets. Yet, these are the cases that matter most for clinical safety.

This data gap leads directly to the second challenge: generalization . Imitation learning inevitably plateaus at the edges of the data distribution. Reinforcement learning (RL) offers a path beyond that plateau. RL can explore millions of interactions, stress-test policies, and learn from failures. To do so effectively, it needs simulation that is realistic enough to produce meaningful policies—and fast and scalable enough to train at scale.

The third challenge is development velocity . Medical robotics development relies on benchtop phantoms, cadaver studies, animal models, and limited clinical evaluations. These remain essential, but they are inherently sequential, expensive, and difficult to scale. Iterating on designs and algorithms currently takes months, pushing total development cycles to 4–7 years .

These three challenges point to the same missing infrastructure: an open, GPU-native simulation framework capable of modeling device–anatomy interactions with the fidelity required for robot training.

NVIDIA Medical Physics Simulation framework—an open source, GPU-accelerated capability within NVIDIA Isaac for Healthcare, was built to fill that gap. Developers can generate anatomical digital twins, simulate device–anatomy interactions and medical imaging, and train reinforcement learning policies at GPU scale—all within NVIDIA Isaac Sim and NVIDIA Isaac Lab.

Medical Physics Simulation provides both classical physics-based solvers and generative world-model-based simulators for real-time simulation of device–anatomy interactions—enabling RL-ready policy training across healthcare robotics segments. Both modules provide a modular research and engineering platform for interactive surgical and interventional simulation, robot learning, synthetic data generation, and procedure development. They are well suited for machine learning workflows because both simulation and learning run on the same GPU, avoiding the overhead of repeated CPU–GPU memory transfers.

The Endoluminal Simulation Module is released for general availability in Isaac for Healthcare. It enables real-time simulation of diagnostic and interventional procedures involving long, flexible surgical instruments navigating through endoluminal cavities. It is implemented as a standalone package, so it can be integrated independently into different workflows and environments. This initial release shows catheter navigation through the vascular system under fluoroscopic guidance.

The module is implemented in Python using NVIDIA Warp and Newton Physics . The flexible instruments are modelled as Cosserat rods, providing a solid theoretical foundation for simulating materials’ bending, twisting, and stretching deformation. To capture the complex nonlinear dynamics of these one-dimensional rods efficiently on the GPU, extended position-based dynamics (XPBD) was selected as the primary simulation method.

XPBD typically relies on local, iterative constraint projections. For long instruments, however, local projections may require many iterations to propagate motion from the controlled proximal end to the distal tip. This module instead assembles the coupled rod constraints into a matrix system.

Each rod segment contributes six constraint equations—three for stretch and shear and three for bending and twisting—producing a block-tridiagonal XPBD system with 6 × 6 blocks. The Thomas algorithm solves this system in linear time with respect to instrument length, while independent instruments are processed concurrently across vectorized environments on the GPU.

This globally coupled solver propagates inputs such as proximal insertion and rotation along the entire instrument within each simulation step, even for very long instruments, enabling immediate response to user manipulations at the distal end.

Figure 1. Catheter insertion and steering through a vascular model

Simulating catheter-vessel interactions

Catheter–vessel interaction is also evaluated in parallel on the GPU. Signed closest-point queries are evaluated directly against the patient-specific triangle mesh, avoiding a separately precomputed distance field. Instrument samples are projected back into the lumen when they penetrate the wall while retaining tangential sliding motion. The current release supports rigid, one-way containment; two-way coupling to deformable vessel walls is planned for a future release.

Control inputs are expressed directly through the rod’s boundary conditions and rest state. Insertion advances the constrained proximal end along the introducer axis, hub rotation sets its orientation, and distal steering modifies the rest curvature of the final rod segments rather than applying an artificial external force. The bend-and-twist constraints then propagate these inputs through the instrument.

The Endoluminal Simulation Module maintains compatibility with other Newton solvers through a common API. Torch–Warp interoperability, provided through wp.from_torch and wp.to_torch keeps the control and reinforcement-learning interface zero-copy and fully GPU-resident. This enables straightforward integration with NVIDIA Isaac Lab, a framework designed for large-scale robot learning. Isaac Lab manages environment orchestration and rollout control, while the solver executes the GPU-based dynamics and returns device-resident tensors for observation, reward, and control loops.

This co-design of physics and imaging supports high-throughput policy training at scale, including experiments with 512 parallel environments and 1,500 training iterations. Reported performance is approximately 1,300 Hz for single-environment physics, 60 Hz for physics simulation across 512 environments, and 63 frames per second for the complete simulation and rendering loop at a resolution of 256 × 256 pixels. The platform also provides a foundation for future policy training directly from fluoroscopic images and other pixel-based observations.

Figure 2. Simulated catheter navigation in patient-specific vasculature

The catheter navigation workflow runs as a unified simulation loop that combines two coupled patient-specific components. The vasculature-digital-twin package generates the anatomical attenuation/geometry artifacts ( mu_volume , vessel mask/mesh, and centerline), and the endoluminal simulation package advances catheter mechanics under XPBD constraints inside that anatomy. For each frame, the solver state is transformed into renderable catheter segments and composited by fluorosim into the same CT-derived imaging domain. This keeps the physics and fluoroscopy synchronized in one end-to-end loop while additional detector-response effects improve the realism of the resulting fluoroscopic images.

Figure 3. Synchronized catheter physics and fluoroscopic imaging

The Surgical Simulation Module available through early access is a real-time GPU-accelerated surgical simulation framework, designed to model soft-tissue deformation and surgical interactions with haptic feedback and rendering within a unified simulation pipeline.

Similar to the Endoluminal Simulation Module, it is also implemented in Python using NVIDIA Warp and Newton Physics . Unlike monolithic physics engines, it organizes its functionality as explicitly ordered simulation systems. Each procedure is configured through the sequence of deformation, collision, grasping, cutting, clipping, diathermy, and rendering operations. This makes the physical GPU pipeline inspectable, configurable, and easier to extend.

Figure 4. Robotic interaction with deformable soft tissue

Soft tissue is represented using tetrahedral meshes. Deformation is solved with position-based dynamics using distance- and volume-preserving constraints. Corrections are accumulated and averaged on the GPU, reducing constraint-ordering artifacts and supporting efficient parallel execution (Jacobi solver).

Additionally, both simulation and visualization loops remain on the GPU, including deformation, collision queries, grasping, thermal diffusion, topology updates, and surface reconstruction. NVIDIA CUDA graph capture further reduces kernel-launch overhead, while direct GPU-to-renderer data transfer (CUDA interop) avoids unnecessary CPU/GPU copies. This enables real-time simulation at more than 30 frames per second with eight physics substeps and eight internal deformation constraint iterations on one consumer-grade GPU.

The initial release of this module supports an end-to-end cholecystectomy procedure (gallbladder removal). The simulation begins with the robotic instruments inserted into the inflated abdominal cavity, visualizing key structures like the liver, gallbladder, and connective fatty tissue. It involves dissection of the hepatocystic triangle, clipping and cutting the cystic duct and artery, and separating the gallbladder from the liver.

Figure 5. Simulated robotic cholecystectomy

Learned world dynamics and generative physics for surgical simulator

The generative approach to medical physics simulation is a complementary technology to physics solvers. Rather than explicitly programming every interaction, a generative world model learns from video and other data how real scenes look and how they change over time. During training, it learns relationships between visual observations, motion, and conditioning signals such as robot actions. At inference time, it uses this learned world knowledge to predict the next video frames.

Because the result is produced in the visual space seen by cameras and robotic policies, the model can reproduce scene detail and appearance that would be difficult to build by hand. It also avoids much of the manual work needed to create every asset, material, and lighting setup for a conventional simulator. The tradeoff is that these predictions are learned from data: they can be visually realistic without providing the explicit physical guarantees of a numerical solver.

NVIDIA Cosmos provides the world foundation models for this approach. Cosmos video models use a flow-based diffusion-transformer architecture with a rectified-flow formulation. During generation, the model iteratively transforms noise in a compressed video representation into a coherent sample from the learned data distribution. A video tokenizer compresses pixel-space frames into this latent representation for efficient modeling and decodes the generated latents back into video. Multimodal conditioning makes the generated future controllable. For medical applications, post-training on domain data adapts the broad visual model before specialized anatomy, devices, procedures, and dynamics.

The Cosmos-H family shows several forms of generative medical physics simulation. Cosmos-H-Surgical-Predict forecasts plausible future surgical video from an initial image and text. Cosmos-H-Surgical-Transfer generates surgical video guided by controls such as depth, edges, segmentation, or blur. Cosmos-H-Surgical-Simulator predicts future frames conditioned directly on surgical-robot kinematics.

Together, these models support use cases ranging from synthetic-data generation and controlled domain transfer to action-conditioned robot training and policy evaluation. They do not replace explicit physics in every setting, but reduce scene-authoring burden and provide realistic, observation-level simulations where complete first-principles modeling is impractical.

Cosmos-H-Dreams extends this approach to interactive simulation. It distills a teacher fine-tuned from Cosmos-H-Surgical-Simulator into a causal, few-step student that autoregressively streams future surgical video in response to robot actions. Served through the FlashDreams inference engine, it turns a high-quality offline world model into a real-time environment with frame generation at more than 30 frames per second. The environment can be controlled by a human operator or queried in closed loop by a learned robotics policy on a single workstation GPU.

Isaac for Healthcare supports both classical and generative simulation to address the complementary requirements of healthcare robotics. To provide user trust and quantitative quality assessment, which are the foundations of any healthcare solution, the physically consistent and deterministically modeled environments provided by classical simulation are essential. Generative simulation, in turn, produces diverse and visually realistic environments at scale that classical simulation cannot, enabling clinical applications in both data synthesis and real-time clinical workflows.

Build patient-specific catheter navigation simulations using GPU-accelerated physics, anatomical digital twins, and simulated medical imaging.

Cosmos-H Dreams enables scalable generation of realistic surgical experiences using world foundation models. Developers can use pretrained models, run interactive simulation, or adapt the framework to new procedures and robotic embodiments.