Author | Wu Yuqing Editor | Tian Zhe
On October 17, a distressing accident occurred on the Xupu Bridge in Shanghai. A flatbed truck first collided with a white passenger car, then veered into the opposing lane and struck four more vehicles in succession, resulting in one death and two injuries.
Following the accident, numerous reconstructions and discussions about the entire incident emerged online. If the white passenger car had not changed lanes illegally, or if the truck had braked earlier, would the outcome have been different?
In the real world, people learn lessons from near-misses they experience firsthand and accidents they witness, thereby improving their driving and decision-making skills.
However, for unmanned vehicles that need to avoid risks through training, studying these issues requires changing conditions and making repeated comparisons, but dangerous situations cannot be recreated repeatedly on real roads.
As autonomous driving evolves, simulation testing has long handled this aspect of work. Today, companies like Baidu and Waymo are using world models to expand the boundaries of virtual test environments, creating "rebirth" opportunities for unmanned vehicles.
This means starting from existing scenarios, altering road conditions or driving actions, simulating different subsequent outcomes, and allowing the system to be tested repeatedly under various conditions.
The challenge is that giving an unmanned vehicle a chance to "start over" cannot merely involve resetting the ego vehicle's actions; it must also ensure that surrounding vehicles, pedestrians, and objects respond reasonably. What kind of world model can support such training?
01 How Can the "Rebirth" Provided by World Models Be Useful?
AI scientist and World Labs co-founder Fei-Fei Li once compared cars to wheeled robots, arguing that autonomous vehicles move like "boxes" in a quasi-two-dimensional environment, where the goal is to avoid collisions as much as possible. In contrast, household robots must actively reach out to perform tasks like washing dishes, cooking, and folding clothes.
From the perspective of operational tasks, she considers cars relatively simple, stating, "Autonomous vehicles are actually quite simple robots." However, Dr. Wang Liang, Chief R&D Architect at Baidu Intelligent Driving Business Group, said: "Autonomous driving has never been 'Easy Mode' for embodied intelligence."
For instance, when discussing the difference between L2 and L4, Dr. Wang Liang cited the documentary *Free Solo*. Assisted driving is akin to climbing with protection, allowing for eye-catching maneuvers, whereas L4 is like free solo climbing, where the huge gap in the middle is difficult to bridge.
When unmanned vehicles enter open roads, they must constantly respond to the unpredictable actions of other vehicles and pedestrians, raising requirements for safety, real-time response, and handling traffic conflicts. In his view, the difficulty lies not only in perceiving three-dimensional space but also in the fact that there is no human driver ready to take over at any moment, leaving the system almost no room for error.
In real-world scenarios, when a vehicle changes lanes, even if there is space nearby, it depends on whether the following vehicle yields. Whether to inch forward or ease off the accelerator first also affects the other party's choices, because multiple vehicles on the road are making decisions simultaneously.
The problem researchers need to solve is how to bring this constant back-and-forth interaction happening on real roads into the virtual world.
Fei-Fei Li categorizes the functions of world models into renderers, simulators, and planners. While driving, the renderer allows the car to see the road conditions, the simulator informs you what happens if you steer left or right, and the planner helps decide whether to turn the wheel to reach the destination or avoid danger.
Dr. Wang Liang agrees with Li's classification of world models and further proposes that autonomous driving requires the coordination of these three functions, with the simulator's ability to reasonably deduce consequences being particularly important.
After the driving system executes an action, the simulated environment changes accordingly, feeding new road conditions back to the system so it can make the next decision. For example, if an unmanned vehicle brakes early, the following vehicle in the virtual road must also decelerate.
If the test merely replays recorded road conditions from real vehicles, the following vehicle will still travel along its original trajectory. In this case, even if the unmanned vehicle avoids a collision with the leading vehicle in the simulation, it is difficult to determine whether that braking action would be safe on a real road.
Currently, the concept of world models is becoming increasingly popular, but not every world model can create a credible testing ground that conforms to the interaction laws of the physical world. For example, Marble from Fei-Fei Li's team primarily focuses on generating and editing three-dimensional spaces.
When world models are used as embodied simulators, the core issue is whether they can faithfully reproduce physical laws and causal relationships of actions, allowing agents to complete full tasks in virtual environments that would otherwise require real-world execution. The realism of the visuals or the novelty of generated scenes cannot substitute for verifying the consequences of actions.
Based on this positioning, Baidu proposed a "2+3" evaluation standard for world models.
The "2" refers to the basic thresholds set for physical AI world models: first, causality, meaning the model must derive the next second from known past events just like the real physical world; second, interactivity, which means that when actions are changed during operation, the environment updates accordingly.
The "3" represents the criteria for judging the usability of world models, including physical accuracy, persistence, and real-time performance.
Physical accuracy refers to how reliable the simulated state is, whether it conforms to world laws, and whether capabilities learned in simulation can be applied back in the physical world. Persistence refers to whether the duration of continuous operation and stability can keep up, and whether objects and details drift over long-running sessions. Real-time performance refers to low latency, ensuring that increases in world complexity do not affect model inference time, thereby achieving high simulation testing efficiency.
In short, if a so-called world model can only replicate a small fraction of real-world road participants and physical laws, the reliability of the test results remains questionable.
02 How to Build a Credible Driving Test Environment?
Baidu's proposal of the "2+3" definition and evaluation standards for world models aims to make the environment truly respond to the driving system. But how can a virtual world be constructed to rival a real-world test environment?
Let's look at two recent clips demonstrated by Baidu:
On the left, an unmanned vehicle drives onto an elevated road, yielding space to a vehicle coming from a side road, which then merges into the main road. On the right, the unmanned vehicle switches to the auxiliary road alongside the elevated highway, reducing the merging space, so the vehicle on the side road stops and waits.
Both clips start from the same location, but the unmanned vehicle takes different paths, and the vehicle on the side road makes different choices.
In the next set of clips, the variable changed is weather. On the same road, with identical initial speed and braking commands, the car stops earlier in sunny weather but slides further in rainy conditions.
Rain alters road reflections, visibility, and lidar returns, consequently changing the outcome of that braking action.
Expanding the lens to an entire trip, Baidu set the Marriott Hotel near Hong Kong International Airport as the common starting point in the virtual environment, assigning different destinations to the test vehicles. Even if the start and end points remain unchanged, adjusting traffic flow from low to high density forces vehicles to handle more interactions, slowing down the journey.
All three demonstrations come from ApolloWorld, the world model recently disclosed by Baidu. Yang Ze, Head of Baidu's World Model team, introduced that Baidu models traffic participants on virtual roads as AI agents with individual goals capable of sensing surrounding changes. These agents decide whether to go or stop based on environmental changes, and their actions influence how the unmanned vehicle proceeds.
However, having reactive vehicles is not enough; the driving system must also see and understand these changes. ApolloWorld generates camera images and lidar point clouds for the same moment's road conditions. For example, the truck in the image below appears in the corresponding position in the point cloud. Information is exchanged between different camera views, as well as between cameras and lidars.
The purpose of multimodal joint generation is to allow different observations to jointly describe the same world before the eyes. As for how these observation frames are generated, ApolloWorld passes the aforementioned states as conditions to the generative model, uses text and other conditions to control weather and lighting, and employs digital twins to provide geometric structure and appearance information for the scene, ensuring that generated frames have clear constraints rather than being imagined out of thin air.
After the world model generates observation frames, the driving model accelerates, brakes, or steers accordingly. Combining the responses of the ego vehicle and other traffic participants forms the next state of the world. Feeding this new state back into the model continues the simulation of changes, allowing traffic scenarios to evolve continuously.
Two sets of models collaborate in the cloud to complete tests, and the iterated driving model is then deployed to vehicles, effectively equivalent to AI teaching AI to drive.
In February this year, Waymo also released a world model capable of responding to driving actions and generating camera and lidar observations. Unlike Baidu's approach, Waymo mainly demonstrated how to change road conditions to generate rare scenarios like floods and tornadoes, without showcasing complete long-distance trips.
Baidu's focus, however, is on how other traffic participants around the ego vehicle respond, and how the system runs through an entire trip from start to finish according to the set destination.
In comparison, Baidu's ApolloWorld is closer to the actual operational process of Robotaxi services, covering verification from order acceptance and passenger pickup to drop-off. Currently, in the cases displayed by Baidu for ApolloWorld, vehicles successfully avoid hazards, showing positive results. However, in the eyes of researchers, failures also have value; the utility of the virtual test environment lies precisely in identifying easily overlooked conditions and trying repeatedly.
It is worth noting that although the test environment can pose questions repeatedly, determining which questions are worth testing still requires answers from real roads. Especially when unmanned vehicles head to a new city, road conditions and development may differ. What can the roads Baidu has previously traveled offer for testing in the next city?
03 When Fleets Expand to More Cities, How Can Testing Resources Keep Up?
Road layouts, traffic flows, and local driving habits vary across different cities. However, the safety boundaries of driving systems must encompass all types of situations, implying increasing validation demands.
Yet, test vehicles and safety drivers are limited, while validation needs grow with business expansion.
Baidu utilizes ApolloWorld to move some tests originally requiring real vehicles, manpower, and road time to the cloud for parallel execution. According to Baidu's calculations, the virtual mileage driven in a day is roughly equivalent to the mileage covered by one real vehicle equipped with a safety driver testing continuously for three years.
The speed at which virtual mileage can be accumulated depends on computing power, but which scenarios to run depends on what situations the vehicle has encountered or might encounter on real roads.
How does the road experience accumulated by Baidu on thousands of Luobo Kuaipao (Apollo Go) vehicles enter the world model?
According to Baidu's disclosure, Luobo Kuaipao has accumulated over 350 million kilometers of autonomous driving mileage, with over 240 million kilometers being fully unmanned. The team identifies issues worthy of reproduction from operations and constructs scenarios combining maps and driving data. By varying traffic density, participant aggressiveness, and driving actions, the same scenario can derive different test conditions.
Dr. Wang Liang stated that Baidu and Tesla are the only two companies globally applying end-to-end technology to scaled L4 autonomous driving. Among them, Baidu's fully unmanned operating mileage for its end-to-end model exceeds one million kilometers.
But after deploying end-to-end models on vehicles, the true test has just begun.
For autonomous driving to achieve scale, it must simultaneously balance safety, intelligence, and scale, among which exists an "impossible triangle."
The existence of ApolloWorld moves part of the training and validation process, which was highly dependent on real vehicles, to the cloud. For Baidu, this means having another method for higher-efficiency safety validation.
Before entering a new city, the team constructs scenarios using local maps and collected data, conducts parallel tests in the cloud, and then hands over issues needing confirmation to real-vehicle validation.
This way, while expanding the scope of validation, the demand for test vehicles and safety drivers does not need to increase proportionally, enabling Baidu's autonomous driving business to rapidly develop new cities and pursue global expansion.
Thus, Baidu's world model is defining new simulation standards and propelling autonomous driving toward its AGI moment.
Whether it is cones or pedestrians suddenly appearing on the road, or the major accident initially mentioned on the Xupu Bridge, it is difficult for researchers to wait for such events to recur in the same place.
But in the virtual world created by ApolloWorld, unmanned vehicles get chances for "rebirth." Even subtle changes can impact the final outcomes for numerous road participants. Countless experiences from virtual worlds ultimately flow into reality, changing the safety and traffic efficiency of people and vehicles in the real world.
In the future, measuring the expansion capability of an autonomous driving company will involve not just fleet size and mileage, but also how quickly it can transform accidental challenges faced by one vehicle into lessons learned by the entire massive fleet.