An end-to-end model may show excellent metrics on an offline test set, but once deployed in a real vehicle, it can expose issues that never appeared during the training phase.
Why do end-to-end models perform flawlessly in training but encounter problems on the road?
Today, we will delve into this topic in detail.
01. Why Does Training Go Smoothly but On-Road Performance Falter?
During the training phase, the model learns the relationship between inputs and driving behaviors from massive amounts of historical driving data. By feeding the model a large number of samples containing information from cameras, vehicle states, and navigation, it can learn the mapping between inputs and driving behaviors.
In the validation phase, another batch of data not used in training is utilized to verify whether the model can maintain the same level of performance.
Although this testing is important, it still represents a relatively typical offline, open-loop evaluation. The model's predictions do not retroactively alter the already recorded data for the next time step.
When the model participates in the vehicle's closed-loop decision-making, regardless of whether it outputs driving intentions, trajectories, or other intermediate results, these outputs ultimately influence the vehicle's subsequent speed, position, and state through vehicle execution.
Consequently, the next input received by the model may differ from the next frame in the original data.
There is a significant difference between these two testing approaches.
Offline testing asks the model how it should handle a specific scenario; real-world driving, on the other hand, observes the final state the vehicle enters after the model makes continuous decisions.
For instance, if the model slightly deviates from the ideal driving trajectory at an intersection, the deviation itself might not be severe. However, once the vehicle's position changes, the lane markings, surrounding vehicles, and spatial relationships of the road observed at the next time step will also change.
If the model generates another slight deviation under this new state, the errors may continue to accumulate.
Therefore, good predictive performance at a single time step or on a single data segment does not automatically equate to stability during long-term continuous driving.
This is also why autonomous driving R&D is placing increasing emphasis on closed-loop testing—allowing the model's decisions to participate in the subsequent evolution of the vehicle's state and environment, and then observing the results after continuous decisions, rather than merely comparing the model's outputs with reference answers in a given dataset.
02. The Model May Enter Unseen States
End-to-end models frequently face the issue of distribution shift.
Training data records driving behaviors that have occurred in the past, whereas the behaviors generated by the model after deployment on the road may not perfectly align with the driving behaviors in the data.
Assume that in the training data, vehicles are mostly in relatively common positions within the lane, and the model learns how to drive under these specific states.
When the model drives autonomously, if a single judgment causes the vehicle to slightly deviate from the normal trajectory, the subsequent inputs it perceives may differ from the typical states in the training data.
At this point, the model is not merely facing an unseen vehicle; rather, it is encountering an unseen combination of vehicle position, road relationships, and traffic states.
This is also what makes end-to-end systems particularly unique.
Traditional modular systems can break down problems into multiple modules such as perception, prediction, and planning. When issues arise, it is easier to attribute potential sources of the problem to different modules.
In contrast, end-to-end models integrate more capabilities into a unified learning system, meaning many issues ultimately manifest directly as abnormal output behaviors from the model.
Furthermore, distribution shifts in real-world driving do not originate solely from the road itself. Weather, lighting, road construction, behaviors of traffic participants, vehicle states, and driving habits in different regions can all cause discrepancies between real-world data and training data. Therefore, having a sufficiently comprehensive training set does not mean all problems will disappear, because real-world roads are not a finite scenario library but a complex system formed by the continuous combination of different environments, participants, and vehicle behaviors.
03. Why Are Rare Scenarios More Prone to Exposing Problems?
Scenarios such as normal car-following, regular intersection traversal, and standard lane changes are more likely to yield a large number of samples in practically collected data.
However, long-tail scenarios, which truly determine the upper limits of the system, are exceedingly rare.
For example, scenarios where construction zones suddenly alter road boundaries, vehicles make abnormal maneuvers, road markings are inconsistent with actual traffic flow, or multiple low-probability factors occur simultaneously are difficult to collect. These scenarios are not necessarily completely absent from the training data; rather, they may lack sufficient variations. This creates a problem: the model may have learned how to handle common scenarios in the training data but has not fully learned how to respond stably after a state shift occurs.
Therefore, the data closed-loop for end-to-end models is not simply about continuously increasing total mileage. Instead, it requires continuously identifying scenarios where the model is truly prone to making errors and reintroducing these scenarios into the training, validation, and testing systems. This is also why autonomous driving R&D increasingly emphasizes long-tail scenarios and hard sample mining. Consequently, the value of simulation technology in training large autonomous driving models is growing. Simulation does not merely allow large models to run more virtual miles; it can actively alter variables such as weather, traffic participant behaviors, and road conditions to construct scenarios that are difficult to collect in large quantities on real roads, and further conduct closed-loop testing.
04. Why Does Flawless Training Not Equal Flawless On-Road Performance?
Training and offline validation focus more on whether the model can produce reasonable results when faced with existing data. Real-world driving, however, focuses on whether the model can consistently keep the vehicle in a safe and controllable state after making a continuous series of decisions. There is a crucial chain between the two: model output → vehicle state change → new environmental input → re-decision → potential error accumulation. As long as a deviation occurs at any step, causing the vehicle to enter a state less covered by the training data, subsequent problems may be further amplified. Therefore, what end-to-end models truly need to solve is not simply achieving higher accuracy on the training set, but gradually narrowing the gap between the offline data distribution and the real-world closed-loop driving distribution.
This is also why today's end-to-end autonomous driving R&D does not end once model training is completed. A continuous data closed-loop must be formed among real-world road data, hard scenario mining, simulation, closed-loop testing, and subsequent training. Thus, the phenomenon where training goes smoothly but on-road performance falters is not a simple defect unique to end-to-end models, but a challenge that learning-based driving systems must face when transitioning from offline data to real-world closed-loop operations. A truly mature end-to-end system must continuously discover under which states the model is prone to failure, and then, through data, training, and closed-loop validation, bring more of these originally failure-prone states into the system's coverage for training, validation, and testing. The true difficulty of end-to-end systems has never been just about training the model well, but about enabling the model to consistently make the right decisions in the real world.