"World models" have become one of the most highly discussed concepts in the field of Artificial Intelligence (AI). From model-based reinforcement learning and video generation to embodied robotics, and further to the grander vision of "Physical AI," researchers across various subfields are building systems they call "world models." Even ordinary consumers are starting to hear this term in various contexts.
However, an awkward reality is that even within the academic community, there is still no consensus on the definition and internal architecture of "world models." Therefore, combining the latest frontier research, this article attempts to answer three fundamental questions: What is a world model? What are its categories? And what can it do?
Origins: An Insight from 1943
World models are not a new concept. As early as 1943, psychologist Craik proposed that organisms are able to survive and reproduce because they construct "working models" of the physical world in their brains.
The key to this insight is that organisms do not need to rely on slow, trial-and-error physical interactions for everything. The brain can internally simulate hypothetical scenarios, predict the outcomes of candidate actions, and pre-compute optimal strategies. This is precisely the embryonic form of today's "world model" concept: running the world through the mind first before taking action.
Today, this concept is widely applied in fields such as embodied intelligence, including autonomous driving and humanoid robots, as well as computer vision.
Classification: Fei-Fei Li's "Functional Tripartite Classification"
Faced with a multitude of conflicting definitions, Fei-Fei Li, known as the "Godmother of AI," recently proposed a functional classification method (Functional Classification of World Models - Fei-Fei Li's Latest Article), categorizing world models into three types based on their output capabilities:
- Renderer: generates videos, "painting" the internal world;
- Simulator: predicts state transitions, answering "what happens next";
- Planner: directly generates actions, guiding agent behavior.
The value of this classification lies in unifying the language of "what a world model can output." However, its limitations must be noted: it only specifies outputs and applications, without answering what is inside a world model, how it operates, or what its architecture looks like. This is precisely the starting point for the next debate.
IV. Route Debate: Pixel Reconstruction or Latent Space Prediction?
Regarding the internal construction methods of world models, there is currently a proliferation of ideas, with two schools of thought being the most representative:
- Fei-Fei Li's World Labs believes that a world model is a unified internal model that can be decoded into RGB pixels, state vectors, or candidate action suggestions depending on different query interfaces—one model, multiple outputs.
- The JEPA school, led by Yann LeCun (World Model V-JEPA Enters ADAS Applications, Ready to Disrupt Physical AI), argues that generative pixel-level reconstruction has never been the right objective: predicting in the latent space is sufficient, as pixel-by-pixel reconstruction wastes precious representational capacity on irrelevant photometric details.
In fact, both schools share a common underlying cognition, which can be analogized to Large Language Models (LLMs): the success of LLMs is essentially the compression of human language knowledge; whereas the core objective of world models is the compression of real-world physical laws (the joint distribution of sensory observations and agent actions). The day a world model succeeds means that the knowledge of the physical world has been compressed into the model.
The focus of the debate can therefore be summarized in one sentence: how should the world be represented, and to what extent should it be compressed? Precise pixel-level reconstruction is not the ultimate goal. The key to the current deployment of world models is to retain sufficient effective information for downstream applications—the difficulty lies in balancing high-fidelity physical details with the abstract semantics of lossy compression.
Data Determines the Upper Limit: Internet Videos Are the Greatest Treasure
The LLM era has already proven that data is the upper limit of AI capabilities. This rule also holds true in the physical world: given fixed architecture and computing power, the upper limit of generalization ability is determined by the diversity of physical experiences represented by the training data; architecture and computing power only affect the efficiency of approaching the upper limit, but cannot raise the limit itself.
The problem is that the data collected by robots and autonomous driving hardware themselves are far from sufficient in scale and diversity. Currently, the only data source that can meet the requirements is the open internet: hundreds of billions of images, videos, and texts. Among them, internet videos are particularly critical. The raw pixel streams in everyday videos implicitly encode the fundamental prior information governing physical reality:
- Object permanence (objects continue to exist after leaving the field of view); the distinction between rigid bodies and deformable materials; kinematic constraints of motion and interaction; temporal dynamics of lighting and shadows; causal logic of occlusion and disocclusion; structured priors of human-like behavior and interaction.
However, this knowledge is "hidden in unlabeled pixels" and cannot be accessed directly. Therefore, the mainstream approach is a two-stage pipeline: first, use automatic filtering and annotation to release implicit physical knowledge from massive web videos to support the upper limit of generalization; then, through refined selection, remove irrelevant visual content and synthesize standardized action representations, retaining only the motion and interaction signals that are physically meaningful for robots, allowing the model to efficiently converge to embodied tasks without losing the extensive physical priors learned from billions of videos. This logic is in the same lineage as the training logic of LLMs.
Formal Definition: One Model, Three Characteristics
Synthesizing the above discussions, a working definition of world models can be given: a world model is a compressed modeling of the state transition process of the physical world, and its construction is constrained by limited computational resources. Under computational constraints, a general physical world model naturally possesses three characteristics:
- Omnimodal: not limited to text or vision, it must be able to model all perceptual modalities to form a unified latent representation;
- Multi-dimensional Asynchronous: in reality, various sensors sample at different frequencies, and the model must be able to handle multi-dimensional, asynchronous (multi-frequency) sequential data;
- Locality: an agent's perception is limited by resources and can only observe local areas, while the external world continuously intervenes on the local area—therefore, the modeling problem is usually formalized as a Partially Observable Markov Decision Process (POMDP).
This definition also brings about a paradigm shift. Most existing models answer the question, "Given the current observation, what will happen next?"; whereas a true physical world model needs to answer, "What is happening, why is it happening, and what will happen?"—this requires cognitive reasoning: identifying the discrete operational states of the system, drawing the boundary between normal and abnormal, and tracing the causal trajectory of the system's evolution to the present.
Why is this important? Because there is an essential difference between the physical world and the digital world: digital data is permanently readable once generated, and language models trained years ago can still answer questions about quantum mechanics today; whereas physical systems are inherently non-stationary and constantly changing, and the most critical events (such as safety failures, rare failure modes) may never have appeared in static training corpora. This is why static world models are destined to struggle in dealing with the physical world—what we need is a self-consistent dynamic system that can autonomously make localized adjustments in real-time for specific applications based on general representations, and remain robust without collapsing when new modalities are integrated.
Two Perspectives: Tools for Understanding vs. Tools for Prediction
There are also two complementary perspectives in the academic community for viewing world models:
As a tool for understanding: compressing sensory data into stable internal representations, revealing the entities, relationships, and mechanisms needed to explain the current situation. Prediction here mainly serves as a training signal, forcing the representation to learn the correct latent structure. Early deep world models (such as Ha & Schmidhuber 2018) followed exactly this approach.
As a tool for prediction: the value depends on foresight capabilities—predicting how the world evolves, generating candidate future scenarios, and supporting planning and decision-making. LeCun's autonomous intelligence framework places predictive world models at the core of reasoning and action, while large-scale video generators like Sora represent this interpretation at the level of the "observable future."
Building on this, World Action Models (WAMs) bridge prediction and action: they can reason with explicit future video (visual planning), with compact latent trajectories, or with structured spatial representations such as optical flow, 3D point flow, and RGB-D trajectories.
On Fei-Fei Li's functional axis of world models, WAMs are primarily planners, but usually also possess simulator functions—because future state prediction is itself a component of action generation, rather than an auxiliary means. If we classify the current mainstream world models, we can organize the following image:
Beyond Robotics: Can World Models Conduct Scientific Research?
The previous discussions have all revolved around everyday physical environments—robots, vehicles, and manipulable objects. Can this framework be transferred to scientific research? The answer is yes, because the basic paradigm remains unchanged, always consisting of three elements: the state of the system, available observations, and implementable interventions.
Applied to various disciplines:
Chemistry: the state is the molecular species, structure, and reaction progress; observations are spectra, chromatograms, and yields; interventions are controlled operations in experiments.
Biology: the state covers multi-scale processes from molecular networks to organisms; interventions include genetic perturbations and drug administration.
Astronomy: the state cannot be directly obtained and can only be indirectly observed through instruments; "planning" is reflected in the selection of targets, instruments, and observation plans.
In fact, existing scientific models have already partially embodied this formulation: GraphCast in the field of meteorology learns an autoregressive transition model of the global atmospheric state for medium-range forecasting—it is a structured scientific simulator, such as predicting the arrival of a current typhoon, but it is not yet an agent world model because it does not select interventions and has not formed a "prediction-action" closed loop; MDGen at the molecular scale learns a generative model of molecular dynamics trajectories, but similarly does not constitute a complete experimental planning system.
A complete agent scientific world model should go even further: not only generating plausible measurement results, but also capturing mechanisms, adhering to physical constraints and invariants, quantifying its own uncertainty, supporting counterfactual interventions, and incorporating these capabilities into the experimental closed loop—proposing hypotheses, predicting outcomes, executing experiments, and updating internal states based on new evidence.
Conclusion
If we summarize what a world model is, we can compress the entire article into three sentences: What is a world model—a compressed modeling of state transitions in the physical world under limited computing power constraints; it does not just predict pixels, but is an internal simulator containing latent states, observation formation, and action-conditional dynamics. How does it grow—data diversity determines the upper limit of generalization, and internet videos are currently the only qualified source of physical priors; of course, it still needs to be trained according to the training methodology of LLMs, which we will share in subsequent articles.
Where is it going—from answering "what will happen next" to answering "what is happening, why, and what will happen"; from robotics and autonomous driving to the closed loop of scientific discovery in chemistry, biology, and astronomy. Finally, follow Vehicle; we will have subsequent articles to deeply explore the key technologies and architectures of world models.
References and Images
- Understanding World or Predicting Future? A Comprehensive Survey of World Models JINGTAO DING∗ , YUNKE ZHANG∗ , YU SHANG∗ , JIE FENG∗ , YUHENG ZHANG† , ZEFANG ZONG† , YUAN YUAN† , HONGYUAN SU† , NIAN LI† , JINGHUA PIAO† , YUCHENG DENG, NICHOLAS SUKIENNIK, CHEN GAO, FENGLI XU, YONG LI, Department of Electronic Engineering, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, China. A Definition and Roadmap for World Models Physical Intelligence Team, Shanghai AI Laboratory
*Reproduction and excerpting are strictly prohibited without permission. How to obtain the reference materials for this article: join our Knowledge Planet to download a massive amount of reference materials from our official account, including the above references.