NVIDIA 解析世界动作模型如何重塑机器人操作

Technical Blog

sessionStorage.removeItem('facets.search.text'); sessionStorage.removeItem('facets.search.text-copy'); sessionStorage.removeItem('facets.search.filters');

World action models (WAMs) replace vision-language model backbones with video world models to give robot policies a physics prior for predicting how scenes evolve. NVIDIA research shows WAMs learn from diverse interaction data, generalize to unseen environments through learned dynamics, and adapt to new robot embodiments with fewer demonstrations than vision-language-action models. NVIDIA Cosmos 3 provides an open omni-model foundation trained on 767 million images, 348 million real-world dynamics videos, and 8 million action samples spanning manipulation, driving, and egocentric motion. Cosmos 3 Policy DROID models post-trained from the 16B Nano and 4B Edge checkpoints jointly predict action trajectories and future camera views while retaining the full multimodal architecture. Deployment options include workstation serving with the 16B Nano model on an NVIDIA RTX PRO 6000 and on-device inference with the 4B Edge model on Jetson Thor at 15 Hz control frequency. Next Steps Explore the Cosmos 3 model collection to access pretrained checkpoints and datasets. Read the Cosmos 3 technical report for architecture details and evaluation results. Join the Cosmos Discord channel to connect with the community and get support.

Powered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. Learn more

A central challenge in robotics is building policies that generalize beyond the demonstrations they’re trained on. A policy that succeeds in a training scene often fails when object shapes, positions, or lighting change. Generalizing to these new conditions requires the policy to understand the tasks underlying physics, not just mimic the demonstrations. This ability comes from the backbone it’s built on.

The standard way to build a language-conditioned robot policy is to add an action module to a pretrained vision-language model (VLM) , producing a vision-language-action (VLA) model. This approach has carried generalist manipulation a long way. But a VLM backbone learns to describe the world, not predict how it evolves . That missing dynamics model is exactly what a robot needs when a task depends on anticipating how a scene will change. A growing line of research replaces the language backbone with a video world model , producing world action model (WAM) . NVIDIA researcher Jim Fan explored this shift in his Robotics’ End Game talk—an idea later summarized as “VLAs are dead, long live World Action Models.”

This post explores how post-training can turn WAMs into specialized robot policy, how WAMs compare to VLAs, and why the open NVIDIA Cosmos 3 world model provides a strong foundation for building WAMs.

In the VLA paradigm, a pretrained VLM provides semantic understanding of scenes and language instructions, while post-training learns to map that understanding to robot actions. Modern generalist robot policies increasingly build on this approach.

However, a VLM is optimized to produce text about images, not to model how a scene will evolve. It does not learn what happens to a mug when the gripper closes, how a towel folds, where an object lands when released. VLAs generalize well semantically but are less effective at physical generalization to unseen behaviors and environments, as their backbones model language and perception rather than world dynamics.

A WAM overcomes that dynamics-modeling limit by building the policy on a video world model. Because the backbone models how the world evolves, post-training does not have to teach dynamics from scratch, it specializes a model that already has a physics prior. The NVIDIA research paper World Action Models are Zero-shot Policies shows that jointly predicting video and action gives a policy properties a VLA cannot easily acquire:

For a team building a policy, these are the practical wins: less data to reach a given capability, better behavior outside the training distribution, and a shorter path to a new embodiment. They are properties of the pretraining backbone , so they show up in every policy post-trained from it.

Cosmos 3 is an omni-model world foundation model built on a Mixture-of-Transformers (MoT) architecture. Multimodal input flows through an autoregressive transformer for reasoning producing discrete tokens such as text. This guides a diffusion transformer for continuous modalities, including image, video, audio, and action. Text is generated by next-token decoding; everything else, including actions, is synthesized through iterative denoising. A single model spans these modalities while keeping the generation mechanism best suited to each. Cosmos 3 comes in three sizes: 4B NVIDIA Cosmos Edge, 16B NVIDIA Cosmos Nano, and 64B NVIDIA Cosmos 3 Super.

What makes Cosmos 3 a strong foundation for post-training is the breadth of its physical-world data. The dataset includes roughly 767M images,348M videos of real-world dynamics, 8M action samples spanning robot manipulation, autonomous driving, camera motion, and egocentric motion.

Figure 1. A VLA generates robot actions from semantic reasoning and robot state, while a WAM jointly predicts actions and future world states

From world model to robot policy: Cosmos 3 Policy DROID models

Cosmos 3 is the starting point for specialization. Cosmos3-Nano-Policy-DROID is a 16B-parameter policy post-trained from Cosmos 3 Nano for the DROID platform, which is a Franka Panda arm with a Robotiq gripper.A 4B version, Cosmos3-Edge-Policy-DROID , is post-trained the same way, which can be used for on-device deployment. Given a language instruction and multi-camera observations, it generates robot action trajectories. Three properties follow from its omni foundation:

Figure 2. A robot observes a banana on the desk (left); the model’s predicted action plan is shown as text

Deployment considerations

A WAM carries the full generative world model, not just an action head. It’s larger than compact VLAs, but Cosmos 3 map to different deployment tiers rather than forcing one trade-off:

WAMs represent a shift from learning to act to learning how the world evolves. Cosmos 3 makes this approach practical:

The fastest way to evaluate whether a WAM beats your current VLA is to post-train one from Cosmos 3 on your own data and compare.