NVIDIA Cosmos-H-Dreams：将实时生成式模拟引入手术机器人

1. From Surgical World Model to Interactive Simulator

Cosmos-H-Surgical-Simulator is an action-conditioned world foundation model built on NVIDIA Cosmos-Predict2.5-2B and post-trained on the Open-H-Embodiment dataset. Given a surgical context frame and a future robot trajectory, it generates video showing the likely visual consequences of those actions. This makes it useful for offline policy evaluation and synthetic data generation. A recorded or policy-generated trajectory can be sent to the model, the corresponding rollout can be generated, and the result can be inspected or scored without repeatedly executing the motion on a physical robot.

Cosmos-H-Dreams moves the model into the real-time regime. Starting from the multi-embodiment surgical priors learned by Cosmos-H-Surgical-Simulator, we specialize the model for da Vinci Research Kit (dVRK) tabletop suturing and distill it into a causal student that generates the scene autoregressively. The released model receives an initial RGB frame and a live stream of robot kinematics, then produces the next chunk of frames before continuing with the following action block.

We have also demonstrated the versatility of Cosmos-H-Dreams by collaborating with CMR Surgical and Cambridge Consultants to integrate it with the Versius surgeon controller, enabling real-time operation on the Versius platform.

2. Distilling Cosmos-H-Surgical-Simulator for Real Time

The key challenge is preserving useful surgical dynamics while reducing the cost of generation. Cosmos-H-Dreams uses a teacher-to-student training pipeline designed for long, autoregressive rollouts.

2.1. A Surgical Teacher

The bidirectional teacher begins from the Cosmos-H-Surgical-Simulator Open-H checkpoint, which uses a unified 44-dimensional action representation. For the released dVRK tabletop model, the dual-arm dVRK action content, consisting of relative end-effector translation, rotation, and gripper state, is mapped into this common representation.

The teacher is then fine-tuned on the JHU dVRK tabletop mixture, including successful demonstrations as well as failure and out-of-distribution episodes such as needle drops, missed throws, and unsuccessful knot ties. These failures are important: a simulator intended to evaluate policies must reproduce the consequences of poor actions, not only ideal demonstrations.

To improve stability during long rollouts, the training process progressively increases the teacher's temporal horizon. We start training on a 12-frame horizon and progressively increase this number until 72 frames. At each horizon bump, we initialize the warmed-up model with pretrained weights.

2.2. Causal Warmup

The teacher's denoising trajectories are first precomputed and cached. A causal student is initialized from the teacher and trained to imitate these cached trajectories. This warmup stage teaches the student to operate with causal attention and a streaming key/value cache before it begins learning from its own generated history.

2.3. Self-Forcing Distillation

Autoregressive models face a familiar problem: during training they may see clean, ground-truth context, while during deployment they must condition on their own imperfect outputs. Small errors can therefore compound over time.

Cosmos-H-Dreams addresses this mismatch with self-forcing distillation . During training, the student rolls forward using its own generated context. Distribution-matching supervision from the frozen teacher then guides those self-generated rollouts toward realistic surgical video. This prepares the student for the same conditions it will encounter during interactive inference.

The resulting model supports few-step diffusion, with as few as two denoising steps per latent frame, rather than the many-step process used by the full teacher. It combines the teacher's surgical priors with the causal structure needed for streaming.

3. FlashDreams: The Real-Time Inference Engine

Model distillation is only part of the real-time story. Cosmos-H-Dreams is served through FlashDreams , an accelerated inference library for autoregressive world and video models.

FlashDreams turns the distilled student into a low-latency streaming system through several complementary optimizations, such as streaming KV cache, CUDA Graph capturing, or model compilation.

Together, these techniques bring the distilled surgical world model fine-tune from the roughly ten-frames-per-second regime of standard Cosmos-H-Surgical-Simulator inference to interactive operation (~160 frames per second) on a single NVIDIA RTX PRO 6000 .

Cosmos-H-Dreams also provides the human-machine interfaces that turn generation into interaction. A browser client can send keyboard commands and receive generated frames over WebRTC. A Meta Quest client can map tracked controller motion into robot actions and display the synthesized scene through WebXR. The same model can also be connected to a learned surgical policy, with generated observations and predicted actions exchanged inside a closed loop.

4. Adapting to Your Own Data

While Cosmos-H-Dreams includes a pre-trained checkpoint for tabletop suturing, the system is designed to be extensible to your specific embodiment. To train a real-time student model for your own dataset, we provide a complete recipe for teacher fine-tuning and self-forcing distillation in our step-by-step guide .

5. What Is Next: Toward Closed-Loop Surgical Physical AI

Cosmos-H-Dreams opens a new frontier for surgical simulation: environments learned from real robot data that are responsive enough to be inhabited.

The immediate next step is to evaluate more than visual quality. A useful surgical simulator must respond correctly to actions, preserve instrument and scene structure over long rollouts, and support conclusions that transfer to the physical robot. This motivates a new family of closed-loop benchmarks: tool-tip reach and pose accuracy, gripper-cycle fidelity, idle stability, counterfactual action diversity, long-horizon drift, and agreement between simulated and real policy outcomes.

Real-time world models can also become active partners in surgical policy development. They can generate rare failures on demand, provide scalable environments for reinforcement or imitation learning, and enable rapid evaluation of new policies without tying every experiment to scarce robotic hardware.

Further ahead, real-time simulation enables a series of downstream applications such as latency-aware telesurgery, where the world model helps maintain a more stable display; or interactive surgical rehearsal, procedure planning, and intraoperative decision support. Cosmos-H-Dreams is a research and development platform, not a diagnostic system, a replacement for intraoperative imaging, or a controller for a physical surgical robot. Yet it provides a foundation for exploring these possibilities safely.

As model fidelity, temporal stability, and hardware efficiency continue to improve, real-time generative simulation can help connect surgeon education, synthetic data generation, policy training, and policy evaluation within one shared Physical AI environment.

Explore the models, data, and runtime behind Cosmos-H-Dreams:

Cosmos-H-Dreams code and examples: GitHub repository

Adapt to your own data: Step-by-step recipe for teacher fine-tuning and self-forcing distillation

FlashDreams: GitHub repository

NVIDIA Cosmos-Predict2.5: GitHub repository

Cosmos-H-Dreams brings action-conditioned surgical world modeling into the real-time loop, creating a new environment for people and policies to practice, explore, generate data, and evaluate what happens next.

Models mentioned in this article 3

nvidia/Cosmos-H-Dreams

Image-to-Video • Updated Jul 24 • 1.41k • 15 nvidia/Cosmos-H-Surgical-Simulator Image-to-Video • Updated Apr 17 • 378 • 21 nvidia/Cosmos-Predict2.5-2B Updated Jun 29 • 7.89k • 174 Datasets mentioned in this article 1 nvidia/PhysicalAI-Robotics-Open-H-Embodiment Updated 16 days ago • 188k • 55 More from this author

56 September 29, 2026 How to Use NVIDIA Warp and MjWarp to Accelerate Robotics Simulation and Learning Workflows 51 September 23, 2026 Community GREAT ARTICLE! \n","updatedAt":"2026-07-27T12:10:05.240Z","author":{"_id":"6a50de528e44ae129fc80cf1","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6a50de528e44ae129fc80cf1/NTjXzIMMnHAwbHxPoApo-.png","fullname":"Unified Mentor","name":"UnifiedMentorTeam","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":2,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.4727756679058075},"editors":["UnifiedMentorTeam"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/6a50de528e44ae129fc80cf1/NTjXzIMMnHAwbHxPoApo-.png"],"reactions":[],"isReport":false}},{"id":"6a675e2a55060c7b2f47eed0","createdAt":"2026-07-27T13:33:30.000Z","type":"comment","data":{"edited":true,"hidden":true,"hiddenBy":"","latest":{"raw":"This comment has been hidden","html":"This comment has been hidden","updatedAt":"2026-07-27T20:28:10.419Z"},"numEdits":0,"editors":[],"editorAvatarUrls":[],"reactions":[]}},{"id":"6a6821d8bfdc02b46c092f95","author":{"_id":"6a68205fdfcc66305626d550","avatarUrl":"/avatars/61a778dbf4e7ec252e833d2001cc193c.svg","fullname":"Wilkinson Jeffery","name":"skjlkjs","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2026-07-28T03:28:24.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is a fascinating leap for surgical robotics. Real-time generative simulation could greatly reduce risks in training and planning. The concept of distilling a large model into a causal student model is clever. If you're also exploring video generation from images, check out the Seedance 2.5 image-to-video tool for creative applications.","html":" This is a fascinating leap for surgical robotics. Real-time generative simulation could greatly reduce risks in training and planning. The concept of distilling a large model into a causal student model is clever. If you're also exploring video generation from images, check out the Seedance 2.5 image-to-video tool for creative applications. \n","updatedAt":"2026-07-28T03:28:24.666Z","author":{"_id":"6a68205fdfcc66305626d550","avatarUrl":"/avatars/61a778dbf4e7ec252e833d2001cc193c.svg","fullname":"Wilkinson Jeffery","name":"skjlkjs","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9047848582267761},"editors":["skjlkjs"],"editorAvatarUrls":["/avatars/61a778dbf4e7ec252e833d2001cc193c.svg"],"reactions":[],"isReport":false}},{"id":"6a686bdcfb540a7ebfae8274","author":{"_id":"6a3cd0fb3af2b0b0921e329a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/1Yp81w2J8OBsuzrHcQoZu.jpeg","fullname":"Elton Williams","name":"eltonwilliams","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false},"createdAt":"2026-07-28T08:44:12.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":">Assessing more than just visual quality is the immediate next step. A practical surgical simulator must enable findings that translate to the real robot, maintain instrument and scene organization during extended rollouts, and react appropriately to actions. Tool-tip reach and pose accuracy, gripper-cycle fidelity, idle stability, counterfactual action variety, long-horizon drift, and agreement between simulated and actual policy outcomes are some of the new closed-loop benchmarks that are motivated by this.\n\nThe progress in surgical robotics is genuinely exciting! 🏥🤖 Being able to train and test advanced vision-language-action systems in realistic simulations without risking expensive equipment or delicate procedures could make development much faster and safer. The level of detail needed for tissues, instruments, smoke, and lighting really shows how challenging this field is. 👏\n\nIt’s impressive to see technology moving beyond simple teleoperation and towards smarter, more capable robotic assistance. 🚀 Better simulation environments can help researchers refine these systems before they ever reach an operating room, which is a win for both surgeons and patients. 😊📱\n\nLooking forward to seeing how projects like NVIDIA Cosmos-H-Dreams shape the future of healthcare. 🌍💙 If these simulations continue becoming more realistic, they could play a huge role in improving surgical training, reducing costs, and ultimately delivering better patient outcomes. Amazing work and an exciting glimpse into what's ahead! 🙌✨","html":" \n Assessing more than just visual quality is the immediate next step. A practical surgical simulator must enable findings that translate to the real robot, maintain instrument and scene organization during extended rollouts, and react appropriately to actions. Tool-tip reach and pose accuracy, gripper-cycle fidelity, idle stability, counterfactual action variety, long-horizon drift, and agreement between simulated and actual policy outcomes are some of the new closed-loop benchmarks that are motivated by this. \n \n The progress in surgical robotics is genuinely exciting! 🏥🤖 Being able to train and test advanced vision-language-action systems in realistic simulations without risking expensive equipment or delicate procedures could make development much faster and safer. The level of detail needed for tissues, instruments, smoke, and lighting really shows how challenging this field is. 👏 \n It’s impressive to see technology moving beyond simple teleoperation and towards smarter, more capable robotic assistance. 🚀 Better simulation environments can help researchers refine these systems before they ever reach an operating room, which is a win for both surgeons and patients. 😊📱 \n Looking forward to seeing how projects like NVIDIA Cosmos-H-Dreams shape the future of healthcare. 🌍💙 If these simulations continue becoming more realistic, they could play a huge role in improving surgical training, reducing costs, and ultimately delivering better patient outcomes. Amazing work and an exciting glimpse into what's ahead! 🙌✨ \n","updatedAt":"2026-07-28T08:44:43.150Z","author":{"_id":"6a3cd0fb3af2b0b0921e329a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/1Yp81w2J8OBsuzrHcQoZu.jpeg","fullname":"Elton Williams","name":"eltonwilliams","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.925989031791687},"editors":["eltonwilliams"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/1Yp81w2J8OBsuzrHcQoZu.jpeg"],"reactions":[{"reaction":"🚀","users":["DragonFace","Nourivex"],"count":2}],"isReport":false}},{"id":"6a6897aba4402b1ff4994633","author":{"_id":"67a4fea096628f7a5a4f8d21","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/_9MyBcbHOkpztwiNml71W.png","fullname":"Kiran Bhattacharyya","name":"kdb6df","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2026-07-28T11:51:07.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"includes a pre-trained checkpoint for tabletop suturing\n\nAwesome article. Was clearly in the works over the past 12 months as evident from the NeurIPS collab with Axel Krieger and others along with the Open-H dataset release earlier this year. The tricky part is how one collects their own dataset of the right size for the complexity of the task. More complex tasks will require larger datasets. Training stability and patterns need to be figured out. ","html":" includes a pre-trained checkpoint for tabletop suturing \n Awesome article. Was clearly in the works over the past 12 months as evident from the NeurIPS collab with Axel Krieger and others along with the Open-H dataset release earlier this year. The tricky part is how one collects their own dataset of the right size for the complexity of the task. More complex tasks will require larger datasets. Training stability and patterns need to be figured out. \n","updatedAt":"2026-07-28T11:51:07.279Z","author":{"_id":"67a4fea096628f7a5a4f8d21","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/_9MyBcbHOkpztwiNml71W.png","fullname":"Kiran Bhattacharyya","name":"kdb6df","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9377602934837341},"editors":["kdb6df"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/_9MyBcbHOkpztwiNml71W.png"],"reactions":[],"isReport":false}}],"status":"open","isReport":false,"pinned":false,"locked":false,"collection":"community_blogs"},"contextAuthors":["lzbinden","javirk1","mtoloui","shuver"],"primaryEmailConfirmed":false,"discussionRole":0,"acceptLanguages":["en","zh"],"withThread":true,"cardDisplay":false,"repoDiscussionsLocked":false,"hideComments":true}"> UnifiedMentorTeam Jul 27 GREAT ARTICLE!

Reply deleted Jul 27 • This comment has been hidden skjlkjs Jul 28 This is a fascinating leap for surgical robotics. Real-time generative simulation could greatly reduce risks in training and planning. The concept of distilling a large model into a causal student model is clever. If you're also exploring video generation from images, check out the Seedance 2.5 image-to-video tool for creative applications.

Reply eltonwilliams Jul 28 • edited Jul 28 Assessing more than just visual quality is the immediate next step. A practical surgical simulator must enable findings that translate to the real robot, maintain instrument and scene organization during extended rollouts, and react appropriately to actions. Tool-tip reach and pose accuracy, gripper-cycle fidelity, idle stability, counterfactual action variety, long-horizon drift, and agreement between simulated and actual policy outcomes are some of the new closed-loop benchmarks that are motivated by this.

The progress in surgical robotics is genuinely exciting! 🏥🤖 Being able to train and test advanced vision-language-action systems in realistic simulations without risking expensive equipment or delicate procedures could make development much faster and safer. The level of detail needed for tissues, instruments, smoke, and lighting really shows how challenging this field is. 👏

It’s impressive to see technology moving beyond simple teleoperation and towards smarter, more capable robotic assistance. 🚀 Better simulation environments can help researchers refine these systems before they ever reach an operating room, which is a win for both surgeons and patients. 😊📱

Looking forward to seeing how projects like NVIDIA Cosmos-H-Dreams shape the future of healthcare. 🌍💙 If these simulations continue becoming more realistic, they could play a huge role in improving surgical training, reducing costs, and ultimately delivering better patient outcomes. Amazing work and an exciting glimpse into what's ahead! 🙌✨

Reply kdb6df Jul 28 includes a pre-trained checkpoint for tabletop suturing

Awesome article. Was clearly in the works over the past 12 months as evident from the NeurIPS collab with Axel Krieger and others along with the Open-H dataset release earlier this year. The tricky part is how one collects their own dataset of the right size for the complexity of the task. More complex tasks will require larger datasets. Training stability and patterns need to be figured out.

Reply Edit Preview Upload images, audio, and videos by dragging in the text input, pasting, or clicking here . Tap or paste here to upload images Comment · Sign up or log in to comment

+65

Models mentioned in this article 3

nvidia/Cosmos-H-Dreams

nvidia/Cosmos-H-Surgical-Simulator

nvidia/Cosmos-Predict2.5-2B

Datasets mentioned in this article 1