Over the past year, world models have appeared with increasing frequency across the entire manufacturing industry.
Companies working on autonomous driving are discussing them, those in embodied AI are discussing them, and chip companies and large model companies are also talking about them. Behind this hype is the industry's recognition of the capabilities of end-to-end models in the digital world, naturally leading to the question: once similar methods enter the physical world, can they solve real-world problems in driving, manipulation, and production?
However, models for the physical world quickly encountered a practical issue: while end-to-end models are becoming increasingly powerful, training data is becoming harder to find. Taking autonomous driving as an example, the industry does not lack conventional road data; what is truly scarce are long-tail scenarios such as construction detours, extreme weather, irregularly shaped vehicles, and sudden jaywalking.
Relying solely on road test collection involves high costs and long cycles, and some dangerous scenarios are even unsuitable for repeated collection. The problem faced by embodied AI is even more direct: different robots, different factories, and different tasks all require massive amounts of specialized data, which is currently extremely scarce. Consequently, world models have been pushed to the forefront.
Recently, at the NVIDIA Physical AI Day during SIGGRAPH 2026, Ming-Yu Liu, Vice President of Research at NVIDIA and Head of Cosmos Lab, shared information related to the Cosmos 3 physical world model.
Based on this presentation, this article discusses why world models are important, what technical approach Cosmos 3 adopts, where it can be applied, and what inspirations it might bring to the automotive and manufacturing industries.
Why Are World Models Important?
The current wave of progress in generative AI largely stems from massive amounts of internet data. Language models have read web pages, books, and code, while image models have seen vast quantities of pictures and videos. The digital world has already accumulated enough content that can be directly used for training.
However, robots in the physical world are not so lucky.
Anyone involved in autonomous driving and robot R&D knows that data in the physical world usually comes from real-world operations. An operator controlling a robot for one hour can obtain at most one hour of data. The collection process may also involve collisions, equipment damage, and even safety risks. If we want a robot to experience a thousand types of warehouses, a thousand types of kitchens, and a thousand types of weather, relying entirely on real-world collection would incur unbearable costs.
Traditional simulation can alleviate some of these issues, but simulation scenarios also need to be built by humans. Where objects are placed, how lighting changes, and what equipment failures occur usually need to be designed in advance. If massive scenarios and long-tail situations still rely on manual configuration one by one, the system will revert to a rule-driven scaling approach, making it difficult to cover the complexity of the real world.
World models offer an alternative approach: using models to directly generate the scenarios robots need to experience, and then having the models simulate what might happen in these scenarios. A warehouse can be varied into different layouts, lighting conditions, cargo, and failure states; a road can be varied into different weather conditions, traffic participants, and hazardous events. Robots or autonomous driving systems do not need to make every mistake in reality; they can learn how to cope in advance. In this way, training robots is no longer entirely constrained by real-world time.
Increasing compute power may simultaneously mean adding new training scenarios, new failure cases, and new test data. The models ultimately deployed on vehicles or robots also have the opportunity to achieve stronger generalization capabilities. A crucial point made in the presentation was: data is no longer just collected; it can also be generated through computation.
How Does Cosmos 3 Work?
Dual-Tower Architecture: One for "Thinking" and One for "Simulating"
Cosmos 3 internally consists of two main components, which can be understood as the one "responsible for thinking" and the one "responsible for simulating."
The first component is the autoregressive tower. It is responsible for understanding scenes, comprehending instructions, interpreting events, and breaking down tasks. For example, after receiving the instruction "take the block out of the drawer and place it on the countertop," the model will first list the task steps: locate the block, move the gripper, grasp the block, lift it, move it to the countertop, and then release it.
The second component is the diffusion tower. It is responsible for generating images, videos, sounds, and actions, as well as simulating how the world might change after executing these actions. After completing the planning, the diffusion tower can generate an internal rehearsal: the robotic arm reaches into the drawer, grasps the block, and then moves it to the countertop.
The model can also generate the sound of the block contacting the tabletop, because sound itself is an important cue for judging contact, collisions, and equipment status. Each token in the diffusion tower can attend to other tokens in the same tower, as well as to tokens in the autoregressive tower, allowing it to understand both linguistic information and visual observations simultaneously.
The two towers each have their own independent MLPs (Multi-Layer Perceptrons), meaning independent parameters, but they share multimodal attention. It is this shared attention that enables language, vision, audio, and actions to align with each other, integrating world understanding and world simulation into a single internal representation.
The model needs to know which object in the scene the word "cup" refers to, what motion trajectory corresponds to "grasp the cup," and how to associate the visual of the cup landing on the table with the contact sound. The importance of this architecture lies here: it connects understanding, reasoning, simulation, and action.
After perceiving the environment, the robot can first plan, then imagine the potential outcomes of the actions, and finally execute.
Decomposing Actions into a Geometric Language
For physical AI, just seeing and speaking is not enough. Cars need to steer and brake, robotic arms need to extend and grasp, and humanoid robots need to coordinate their bodies and hands. Actions must enter the model internally, existing at the same modeling level as text and vision. The problem is that the physical bodies of different robots vary greatly. A car has no hands, a single-arm robot has no legs, and a humanoid robot is much more complex than a standard robotic arm.
Cosmos 3 uses a set of geometric action primitives to describe different embodiments, primarily including the position and orientation of the body, the spatial position of the hands or grippers, and the grasping state of the hands or grippers. Using this method, a car can be represented by its body pose; a single-arm robot can be represented by an end-effector and a gripper; a humanoid robot adds body pose, hand positions, and hand states.
Although the physical structures differ, the actions can be transformed into a common geometric representation.
This action language is anchored by the visual structure of the world, connecting pixels, motion, space, and control. Different robots can use the same action representation, giving the model the opportunity to transfer experiences learned from one embodiment to others. Human first-person perspective videos can also be incorporated into this representation. A person's direction of movement, hand positions, and grasping states can all be converted into the action information needed for robot training.
In this way, a massive amount of videos that originally contained only visuals also have the opportunity to become robot learning data.
Seeing this, one might easily think of the frequently discussed VLA, which stands for Vision, Language, and Action models. A brief explanation is needed here. The "L" in VLA usually refers to natural language, while the geometric action primitives in Cosmos primarily belong to "A," which is action representation.
But from a broader perspective, models do need to establish a commonly understood "language" for different robots. This language does not necessarily have to be the natural language used by humans in daily life; it could also be a set of shared geometric symbols, action tokens, or internal representations. The key is that vision, language, and actions can ultimately be aligned within the same representation space.
How Do World Models Interact with Reality?
Cosmos 3 integrates three types of capabilities into a single model: forward dynamics, inverse dynamics, and policy models. When training Cosmos, NVIDIA models actions and world states together, allowing action information to flow bidirectionally.
Forward Dynamics: What Happens After Executing an Action?
Forward dynamics answers the question: given the current state and an action, what will the future state be? In other words, before an action actually occurs, the model first deduces the potential consequences of the action. For autonomous driving, before a vehicle executes a lane change, turn, or brake, it can first predict how the surrounding traffic relationships will change. For robots, before a robotic arm actually moves, it can first judge whether the action will successfully grasp the object or potentially collide with the edge of a drawer.
Inverse Dynamics: What Action Caused the Current Change?
Inverse dynamics answers the question: observing continuous visual states, what action caused the current change? This capability is highly practical because it can convert human demonstration videos or robot logs into training data with action labels. Videos that originally contained only visuals, after being processed by inverse dynamics, have the opportunity to yield action sequences paired with the visuals. A massive amount of driving videos, robot recordings, human first-person perspective videos, and factory surveillance data can thus be repurposed.
Policy Model: What Should Be Done Next?
The policy model answers the question: given the current observation and task goal, what action should be taken next? And after executing the action, what visual results will be produced? It focuses on how the robot completes the task and what consequences these actions will bring. Forward dynamics seeks the results of actions, inverse dynamics seeks the causes behind the results, and the policy model decides the next action. Once these three capabilities are unified into a single model, world modeling is no longer just an upstream tool for robotics; the world model itself begins to enter robotic systems.
Where Can Cosmos 3 Be Applied?
First, Generating Training Data
This is currently the most direct application. Enterprises can use Cosmos 3 to batch-generate factories, warehouses, roads, and robot manipulation scenarios, altering the objects, lighting, weather, camera positions, and failure types within the scenes. Dangerous situations that rarely occur in the real world can also be repeatedly generated and tested within the model. For autonomous driving, it can supplement long-tail data such as construction detours, severe weather, irregularly shaped vehicles, and sudden traffic incidents; for robots, it can generate different workstations, different objects, and various modes of manipulation failure.
Second, Understanding Videos and Automatic Annotation
Cosmos 3 can also analyze existing videos for automatic data annotation. Once a robot recording is fed to the model, it can identify the different stages of a task, explain what happened in each stage, and attempt to recover the corresponding actions. This means that the massive amount of videos and robot logs accumulated by enterprises in the past do not necessarily have to be annotated segment by segment manually. The model can first complete event segmentation, content interpretation, and action recovery, followed by manual review.
Third, Predicting Potential Outcomes of Actions
Before a vehicle actually changes lanes or brakes, or before a robotic arm actually grasps, the world model can first simulate the potential outcomes of the actions. Such capabilities can be used for closed-loop simulation, policy evaluation, and safety testing, and can also help autonomous driving and robotic systems reduce some dangerous trial-and-error in reality.
Fourth, Training Driving and Robot Policies
Cosmos 3 integrates vision, language, and actions into a single model, capable of generating the next action based on current observations and task goals, providing foundational capabilities for end-to-end driving and VLA models. Once the foundation model is completed, enterprises can also use their own scenario data for post-training, and then deploy the capabilities to the vehicle or robot edge through distillation and compression. In other words, it could ultimately be distilled into algorithms that can run on the vehicle or robot edge.
Edge, Nano, and Super: From Foundation Models to Edge Deployment
In the presentation, Ming-Yu Liu shared the three model specifications currently offered by Cosmos 3. Edge has 4 billion parameters, or 4B, focusing on real-time on-device operation. Nano has 16 billion parameters, or 16B, suitable as a starting point for enterprise development and post-training. Super has 64 billion parameters, or 64B, pursuing higher understanding and generation quality.
Edge targets the robot and vehicle edge, Nano targets development and construction, and Super targets higher model capabilities. The three models share the same foundation, which is what we commonly refer to as a foundation model. Many companies hoping to master the closed loop of hardware, software, and data should be developing their own foundation models. Some enterprises, represented by Tesla, are often described by the outside world as wanting to become the "Apple of the robotics industry," and various domestic new forces also want to do the same. To achieve this, mastering hardware alone is not enough; they also need to possess their own data, models, training systems, and edge deployment capabilities.
Competition in physical AI cannot be judged solely by parameter scale. Whether the model can be deployed in a car or robot, how much power it consumes, and whether the response speed is fast enough are equally important. During the presentation, Ming-Yu Liu also introduced a distillation scheme. Generating high-quality visual results with Cosmos 3 originally required about 35 to 50 denoising steps. After distillation, the denoising steps can be reduced to 4 steps. NVIDIA stated in the presentation that an overall acceleration of approximately 25x can be achieved while maintaining quality close to that of the teacher model.
It is necessary to distinguish two concepts here. The reduction of denoising steps from about 50 to 4 is related to the overall inference speed improvement, but the 25x acceleration cannot be directly derived by dividing 50 by 4. The final speed is also affected by the model architecture, hardware, and execution efficiency. A more accurate understanding is: after distillation, the denoising steps are reduced to 4, and the overall acceleration reported by NVIDIA is approximately 25x. For robots and cars, such acceleration has more practical value than simply improving image quality. Physical devices have explicit requirements for response speed, power consumption, and stability, which is also an important research direction for deploying current world models to the edge.
Conclusion
NVIDIA's Cosmos 3 demonstrates a very clear roadmap: robots first understand the world, simulate several possibilities internally, and then decide how to act. Its value lies not only in generating videos. More importantly, it attempts to place text, vision, sound, and actions into the same model and the same token space, allowing training data, environment simulation, task planning, and robot control to use the same world representation. If this approach proves effective, training robots in the future may no longer require re-experiencing every single situation in reality.
Massive amounts of practice will first occur within the model, with real robots responsible for verification and error correction, before sending new experiences back to the model. The same applies to autonomous driving. Long-tail scenarios encountered by vehicles on real roads can enter the world model to be understood, expanded, and regenerated; after the model completes more training internally, it returns to the actual vehicle for verification.
This is probably the most noteworthy aspect of world models: it begins to endow machines with a very preliminary ability to think about what might happen before actually taking action. Finally, a small advertisement: Want to buy a book? About autonomous driving, authoritatively published by Vehicle in conjunction with China Machine Press, for high-paying positions in the automotive industry, an essential book for future embodied AI.
References and Images
- World Models: The Frontier of Physical AI | Physical AI Day at SIGGRAPH 2026 pdf, Ming-Yu Liu, Vice President of Research at NVIDIA and Head of Cosmos Lab
*Strictly prohibited to reprint or excerpt without permission - To obtain the reference materials for this article: join our Knowledge Planet to download massive reference materials from our official account, including the above references.