Author | Xiao En, Editor | De Xin
If two years ago, world models were merely a slightly cutting-edge technical route in autonomous driving, today they have secured two crucial positions in the field: one is the model approach, where world models are widely regarded as the technical path leading to the ultimate goal of autonomous driving; the other is as a vital tool in training, simulation, and long-tail scenario validation.
Real-world road testing can capture a massive amount of daily scenarios, yet it struggles to consistently cover those low-frequency, high-risk extreme cases, such as glare at intersections during rainstorms at night, temporary detours in construction zones, or pedestrians suddenly appearing from behind large vehicle occlusions. Relying solely on fleet "luck" is too inefficient, and manually building scenarios with traditional simulation struggles to align with the complexity of real-world roads.
World models have been pushed to the forefront amid such contradictions.
However, as the concept has gained popularity, its boundaries have begun to blur.
Some refer to autonomous driving simulation as world models, others call 3D scene generation world models, and still others use the term for algorithmic models in autonomous driving. None of these statements are wrong, but they are clearly not referring to the same thing.
Since founding World Labs, Fei-Fei Li has been emphasizing spatial intelligence. Her fundamental premise is that language models do not represent the entirety of how humans understand the world. Recently, in an article titled "A Functional Taxonomy of World Models," she attempted to clarify the concept of world models for the public.
She categorizes world models by function into renderers, simulators, and planners.
The renderer is responsible for generating visuals, the planner for outputting actions, and the simulator in between takes on the expression and deduction of the world state.
A world model needs to know where the vehicles are, where the pedestrians are, how occlusion relationships change, and what the environment will look like after an action occurs. Relying solely on pixel similarity in 2D videos allows the model to learn many visual patterns, but it struggles to consistently learn spatial structures, dynamic trajectories, multi-view relationships, and the consequences of actions.
This leads to another point: to train a truly usable simulator, a world model cannot rely solely on videos that "look realistic." It requires stronger structural supervision.
Enabling the model not only to generate visuals but also to learn the world state behind those visuals.
4D Ground Truth Supervision: World Models Must First Align the World
If a world model is understood merely as video generation, it would be much simpler.
Feed the model a sequence of historical frames, and let it predict the next frame or generate a video for the next few seconds. The more natural the visuals and the more coherent the motion, the more the model appears to "understand the world."
But autonomous driving and robotics require far more than this.
The world models they need cannot merely predict what the camera will see in the next second. They must also know what is happening behind the visuals: where the vehicles are, where the pedestrians are, road boundaries, occlusion relationships, and whether the environmental state will change after a specific action is executed.
They require not only 3D spatial information but also temporal changes. 3D space plus time constitutes the 4D world state.
How to train a world model using 4D data? 4D ground truth supervision is one approach.
Here, "ground truth" can be understood as the standard answer used for comparison during model training. In ordinary video training, the answer is the next frame image; in 4D ground truth supervision, the answer is not just an image, but a dynamic set of world states.
This set of states includes at least several layers.
At the spatial level, it must have 3D geometry. Vehicles, pedestrians, roads, and obstacles cannot remain confined to 2D images; they must have real spatial positions. Which lane the vehicle is in, how far it is from the ego vehicle, its orientation, whether a pedestrian is standing on the roadside or has entered the lane—all these must be expressed.
At the semantic level, it must know what each object is. Vehicles, pedestrians, curbs, traffic lights, and drivable areas are not just pixel blocks, but traffic elements with categories and functions.
At the dynamic level, it must preserve changes over time. Whether a vehicle is accelerating or decelerating, whether a pedestrian is waiting or crossing, and whether two traffic participants are forming a conflict relationship.
If it further enters the decision-making and control stages, the world model also needs to understand the consequences brought by actions.
How the surrounding traffic will change after the autonomous driving system decelerates, changes lanes, or detours; whether the pose of an object will change after a robot grasps, pushes, or collides with it—these are all things the world model must learn.
Not just the visual surface, but also spatial structures, temporal continuity, and state changes.
In autonomous driving, common industry practices generally fall into several categories.
One category uses 3D occupancy or BEV (Bird's Eye View) to express the world state.
Occupancy can be understood as dividing the space around the vehicle into many small grids, with each grid recording whether there is an object and what it is. BEV is more like overlooking the entire traffic scene from above, projecting lanes, vehicles, pedestrians, obstacles, and drivable areas into a unified coordinate system.
Many world models are trained in this state space. Instead of directly predicting the next video frame, they predict the occupancy, BEV, or object trajectories for the next few seconds. In other words, the model answers not "what the next frame looks like," but "what state the world will be in at the next moment."
The advantage of this representation is that it structures the world state, making it easier for the model to learn spatial relationships and dynamic changes. However, it also has obvious limitations. Whether occupancy or BEV, they are essentially a compressed representation of the world.
During the compression process, a large amount of visual detail is lost, such as textures, lighting, weather changes, and subtle information in some long-tail scenarios. For planning and prediction tasks, this information may not be critical; but if the world model is expected to simultaneously handle data generation, simulation validation, and even general physical world modeling, relying solely on occupancy or BEV is often insufficient, as they struggle to fully express the complexity of the real world.
Another approach is to first reconstruct a renderable 3D scene.
Technologies like NeRF and 3D Gaussian Splatting have appeared frequently in autonomous driving simulation and data generation in recent years. Their approach is not to store the real world as video, but to transform the scene into a 3D representation that can be re-rendered from different viewpoints.
This is crucial for world models.
Because autonomous driving training often requires multi-view consistency. If the front camera sees a vehicle, the side camera should also see the same vehicle at the corresponding position. 3D reconstruction is equivalent to building the world first, and then generating data from different viewpoints within this world.
But 3D reconstruction only solves part of the problem. Real roads are not purely static models, so the industry models static backgrounds and dynamic objects separately: roads, buildings, trees, and curbs belong to the relatively stable background; vehicles, pedestrians, and cyclists require tracking their positions and poses over time.
A traffic scene is no longer just many frames of images, but a world updating over time: roads, buildings, and infrastructure form a relatively stable spatial framework, within which vehicles, pedestrians, and cyclists move continuously, while relationships such as occlusion, interaction, yielding, and conflict change continuously over time.
So what role does 4D ground truth supervision play in training?
First, it constrains geometric relationships.
The same vehicle cannot be 20 meters away from the ego vehicle in the front view and appear to be 5 meters away in the side view; a pedestrian cannot be suddenly far or near in space just because the texture changed during image generation. Geometric ground truth constrains object position, scale, and orientation.
Second, it constrains time.
Vehicle trajectories must be continuous, and speed changes must be reasonable; a pedestrian cannot be on the roadside one second and jump to the center of the lane the next without any motion process. Temporal ground truth enables the model to learn "change" itself, rather than just learning frame by frame.
Third, it constrains multi-views.
Autonomous driving does not see the world with a single eye. Front view, side view, rear view, bird's-eye view, and point clouds must ultimately correspond to the same real-world scene. Multi-view ground truth lets the model know that different sensors are seeing the same world, not several unrelated videos.
Finally, it also constrains semantics and action consequences.
Red lights and green lights are not color differences, but traffic state differences; lane lines are not white textures, but part of the driving rules.
Together, these constraints enable the world model to move from a renderer to a simulator.
With 4D ground truth being so important, a question naturally arises: where does the ground truth data come from?
The most traditional method is to collect road test data from the vehicle end.
The vehicle itself collects camera, radar, LiDAR, and positioning data, which then undergoes calibration, fusion, reconstruction, and annotation to obtain 4D ground truth.
However, vehicle-end data inherently carries the limitations of a single-vehicle perspective. It sees "the world in front of this vehicle," which cannot necessarily restore the traffic relationships of the entire intersection. Pedestrians blocked by large vehicles, vehicles approaching rapidly from the side or rear, and conflicts forming on the other side of the intersection are often incomplete.
To train a more reliable world model, the ground truth itself must become more complete.
Is it possible to construct a more complete set of 4D ground truth from perspectives outside the vehicle end, and then transform it back to the vehicle-end perspective truly needed by autonomous driving models?
The answer is yes, and it has already been achieved.
Tangyuan Technology's Approach: Transforming the Roadside World Back to the Vehicle-End Perspective
This question has a specific answer in a public paper co-authored by Tangyuan Technology.
Founded in 2024, Tangyuan Technology is currently headquartered in Xiangcheng District, Suzhou. This location is highly specific. In recent years, Xiangcheng, Suzhou has been promoting intelligent connected vehicles and the integration of vehicles, roads, and the cloud. Cameras, LiDARs, millimeter-wave radars, and edge computing devices have been deployed at numerous intersections and along roads.
These devices originally served vehicle-road coordination and smart transportation, and they also provide a special data source: not the world seen in front of a single vehicle, but a more complete traffic state at an intersection or along a road section.
Tangyuan Technology's CEO, Ren Dongchun, has long been engaged in research on robotics, autonomous driving, and world modeling, and later led algorithm R&D and engineering implementation at Meituan Autonomous Driving. This experience from research to the industrial frontline has also led Tangyuan to focus more on practical issues such as data production, ground truth construction, and training closed loops when developing world models.
With roadside data available, the next step is the methodology.
A public paper by the Tangyuan Technology team, I2V-GS, discusses this exact issue. The full title of the paper is "Infrastructure-to-Vehicle View Transformation with Gaussian Splatting for Autonomous Driving Data Generation." Simply put, it transforms the roadside perspective into the vehicle-end perspective to generate autonomous driving training data.
Roadside devices are mounted high and have a long line of sight, making it easier to see the overall traffic relationships at an intersection; vehicle-end sensors are attached to the vehicle body, positioned low with a narrow field of view, seeing the inputs that the driving system actually faces.
The difference between the two is not just a simple change in camera position, but a complete change in spatial relationships. The same vehicle or the same traffic light will have different sizes, positions, and occlusion relationships in the two perspectives.
Therefore, transforming from roadside to vehicle-end cannot rely on simple image deformation. It must first restore the spatial relationships.
The first step of I2V-GS is to build the scene using 3D Gaussian Splatting.
3DGS uses a large number of Gaussians distributed in 3D space to represent the scene. Each Gaussian carries information such as position, scale, color, and opacity. When combined, the scene can be re-rendered from a new camera position.
The key is not to generate a new image, but to build the scene itself first. Only by placing elements like vehicles, pedestrians, roads, and traffic lights back into a unified 3D space does the subsequent perspective transformation make sense. Otherwise, roadside-to-vehicle-end transformation is merely image-level stitching and completion; it might look realistic enough, but the underlying spatial relationships may not be correct.
However, the roadside data itself is not perfect. Cameras are fixed, have limited fields of view, and their quantity cannot be infinite. Reconstructing a stable 3D scene from these sparse viewpoints is inherently difficult.
This leads to the second key point: depth.
Monocular depth estimation can tell the model what is far and what is near, but the numbers may not be accurate. What looks like 20 meters might not actually be 20 meters. The approach in I2V-GS is to use LiDAR to calibrate the monocular depth.
Here, LiDAR acts more like a spatial reference system to calibrate the true distances in the scene. How far the vehicle is from the intersection and how close the pedestrian is to the lane cannot be guessed solely from images. Once the depth is locked, the subsequent perspective transformation has a more reliable geometric foundation.
Next is adaptive depth warping.
Switching from the high-angle roadside perspective to the low-angle vehicle-end perspective, nearby vehicles change significantly while distant backgrounds change less; areas previously occluded will be revealed, and previously visible areas might be blocked. Directly projecting the image over often leaves holes and distortions.
Adaptive depth warping utilizes depth information to transform the original roadside images into a batch of new pseudo-views. It can be understood as adding training angles to 3DGS: originally having only a few fixed roadside cameras, now through depth guidance, a batch of intermediate views closer to the vehicle-end position is supplemented.
However, after warping, there will still be gaps in the images.
This is where diffusion models are introduced.
It is responsible for filling in missing areas in the new views, such as the back of the vehicle body, the sky, or occluded parts. But the problem with diffusion models is that the content they generate may not be consistent with the real world. If each view is completed independently, different views might not align with each other.
Therefore, I2V-GS does not let the diffusion model generate freely, but constrains the generation process through a cascade strategy and cross-view information, using information from multiple views to cross-validate the completion results.
Simply put, the diffusion model is responsible for filling in missing areas, but ultimately must pass checks for geometric relationships and multi-view consistency.
This is also the most noteworthy part of this paper.
Tangyuan's method is not simply generating vehicle-end video from roadside video, but building a complete data pipeline: first reconstructing the 3D scene, calibrating depth and scale with LiDAR, then supplementing intermediate views via adaptive depth warping, utilizing diffusion models to fill missing areas, and finally using cross-view constraints to ensure the consistency of the entire scene.
In this pipeline, generative models are certainly important, but they are not the only protagonists. More important are the preceding geometric reconstruction, depth calibration, and the subsequent multi-view consistency checks. Without these constraints, the more realistic the generated vehicle-end images look, the easier it is to mask errors.
I2V-GS provides a new approach for building world models. Once the method is established, how can it be transformed into a platform that can continuously produce data and serve customers' training and validation processes?
Real Roads, How to Turn Them into Deliverable Data Products?
For real customers, what they need is data that can be stably produced and used for training. Vehicle-end perspective videos, multi-sensor ground truth, scene annotations, reproducible corner cases, and simulation environments that can repeatedly validate algorithmic strategies.
The Yootta data platform launched by Tangyuan Technology addresses exactly these needs.
Instead of building a fictional scene from scratch and trying to make it as close to the real world as possible, it starts from real roads. Intersections, roads, vehicles, pedestrians, and roadside sensor data are first reconstructed into digital space, and then vehicle-side perspective training data, synthetic videos, 3D ground truth, and simulation scenes are generated from this space.
The industry has the concept of Sim2Real, while Tangyuan Technology's approach is a Real2Sim2Real pipeline.
Real2Sim first moves the real world into digital space; Sim2Real then generates data from this digital world that can feed back into real model training.
The advantage of this is that the starting point of the data is real roads. The spatial structures, object relationships, occlusions, and dynamic changes in real traffic are preserved first, before performing perspective transformations, weather and lighting changes, and long-tail scenario expansions.
But real-world reconstruction is not a panacea.
No matter how many roadside sensors there are, there will be blind spots. For example, the sky, distant backgrounds, occluded areas, and lighting changes in extreme weather are all difficult to fully complete relying solely on 3D reconstruction. If the scene needs to be further generalized to more weather conditions, more lighting scenarios, and more complex environments, reconstruction alone is not enough.
This is also the reason why Tangyuan introduced NVIDIA Cosmos.
Here, Cosmos does not replace 4D reconstruction, but fills in the parts still missing after reconstruction. It can perform missing information completion, diverse view generation, and complex environment generalization, expanding a real intersection into more training samples.
Reconstruct real roads into a 4D world with structural constraints, and then let generative models complete and expand within this scope.
This is the difference between the Yootta data platform and ordinary video generation tools.
It delivers data assets that can enter the autonomous driving R&D process. For example, high-fidelity synthetic video sets can supplement visual training samples; 3D ground truth can be used for perception and scene understanding; multi-scene annotations can be integrated into customers' existing data management and training processes.
On this basis, Tangyuan has also extended its capabilities to closed-loop simulation.
Synthetic data solves "what the model has not seen yet"; closed-loop simulation solves "what the model will do when it encounters it." For the same dangerous intersection, traffic flow density, weather, pedestrian trajectories, and ego vehicle strategies can be changed; the same corner case can also be repeatedly tested across different algorithm versions.
This set of capabilities has already begun to be implemented in real customer scenarios.
Tangyuan Technology has co-built a closed-loop simulation platform based on 4D structural data with Mercedes-Benz and Tsinghua University for autonomous driving development, testing, and validation. In addition, it has also delivered data products such as high-fidelity synthetic video sets, 3D ground truth, and multi-scene annotations to leading OEMs and Tier 1 customers, supporting the use of mass production pipelines.
Conclusion
On real roads, various corner cases happen every day.
A pedestrian stepping out from behind the occlusion of a large vehicle, a sudden lane change by a non-motorized vehicle, a temporary detour at a construction intersection, glare and misjudgment on a rainy night. For ordinary traffic systems, they are just fleeting moments continuously flowing on the road; for autonomous driving, these moments are invaluable training materials.
However, in the past, these materials were difficult to truly utilize.
What Tangyuan Technology has done is to reorganize these scenarios happening in the real world through roadside data, 4D reconstruction, and generative completion, turning them from one-off road events into scene assets that can be repeatedly trained, reproduced, and validated.
The significance of this for the industry lies not only in providing an additional synthetic data method, but it may also change the way autonomous driving acquires long-tail data.
Here, world models are no longer just generating a video, but returning the complexity of real roads back to the training system.