An autonomous driving model may show perfect metrics in offline testing, so why does it still suffer from inaccurate judgments and irrational decisions at complex intersections?
In recent years, end-to-end autonomous driving has seen continuous increases in model scale, training data, and computing power investments, but these investments do not automatically translate into better driving performance. Whether the model can understand complex traffic relationships and handle scenarios underrepresented in the training data depends on what it has learned and how the R&D team verifies the capability improvements.
For the autonomous driving industry, it is not enough to merely pursue larger models; it is more critical to figure out exactly how model scale, data investment, and training methods can be translated into verifiable driving capabilities on real roads.
1. Why Do Larger Models Not Necessarily Mean Stronger Capabilities?
An increase in model parameters means the model has a larger capacity to learn more complex features and relationships. However, a larger model capacity does not equal an inevitable improvement in actual learning outcomes. Whether the training data is sufficient, whether the training objectives are reasonable, and whether the model can transfer the learned patterns to new scenarios all affect the final performance. When following a car on common roads, the traffic relationships around the vehicle are relatively stable, allowing the model to learn vehicle motion patterns from a large number of similar scenarios.
However, at complex intersections, other vehicles may suddenly cut in, pedestrians may change their direction, and multiple traffic participants may simultaneously affect the vehicle's decision-making. The system needs to consider not only what is currently happening but also the potential consequences of different actions. If common driving scenarios account for a high proportion of the training data while complex interactive scenarios are underrepresented, even if the model scale continues to grow, it cannot guarantee an equivalent improvement in decision-making capabilities in complex scenarios.
The model may perform better in familiar scenarios but struggle to cope with situations where the training distribution is insufficiently covered or interactive relationships change. This is also where the relationship between model scale and driving capability is easily misunderstood. A larger model capacity can indeed provide conditions for learning complex driving patterns, but it cannot guarantee that the model has definitely mastered these patterns, let alone ensure proportional improvements across all tasks and scenarios.
In other words, while model scale is important, it is not an independent metric for measuring autonomous driving capabilities. Whether the model can handle complex traffic relationships and make rational decisions in uncertain scenarios is the ultimate question that scale investments need to answer.
2. Does More Data Necessarily Mean Fewer Problems?
Data is a crucial foundation for the capabilities of autonomous driving models, but the total volume of data does not directly represent the amount of effective training information. Suppose the training data contains a large number of high-speed cruising and routine car-following clips; these data help the model learn daily driving patterns but may not sufficiently cover scenarios such as construction detours, complex intersection interactions, or special obstacles. If the continuous increase is mainly in similar data, the model's performance in common scenarios may continue to improve, but the improvement for underrepresented scenarios may be limited. In 2025, a research team published an end-to-end autonomous driving study at the CVPR Workshops, comparing different training data scales ranging from 16 hours to 8,192 hours, and evaluating the model's open-loop metrics and closed-loop simulation performance respectively.
The study found that under the model architecture and simulation settings they adopted, open-loop metrics improved as the data scale increased, but closed-loop driving performance plateaued earlier. This indicates that under specific experimental conditions, improvements in offline metrics do not necessarily translate into proportional closed-loop benefits.
In 2026, another imitation learning end-to-end driving study published at ICRA analyzed approximately 4 million driving demonstration clips, covering 23 categories of scenarios and accumulating over 30,000 hours. The study similarly found that the performance in open-loop evaluations and closed-loop evaluations was not entirely consistent. The study also proposed that, under its test settings, appropriately increasing long-tail scenario data could improve the model's performance in corresponding scenarios. These studies do not imply that increasing data is valueless; rather, they demonstrate that the effect of data growth depends on what the data covers and how the model is evaluated.
Continuously accumulating common scenarios and targeted supplementation of difficult scenarios may yield different benefits; long-tail data is not inherently effective either, as sample quality, scenario context, and training methods still need to be considered. Therefore, autonomous driving R&D needs to establish a closed loop from problem discovery to data supplementation, model training, and effect review.
The R&D team needs to start with failure cases, analyze the relationship between problems and environment understanding, behavior prediction, decision planning, or vehicle execution, and then determine what data should be supplemented, what training objectives should be adjusted, and what tests should be used to confirm that the problem has been improved. Compared to merely pursuing data volume growth, this capability of continuous iteration around specific problems better reflects the engineering value of the data system.
3. Do Better Offline Evaluations Necessarily Mean Improved Driving?
In model capability evaluation, there is another critical issue: do improvements in test metrics truly represent enhanced driving capabilities? In open-loop evaluations, R&D personnel evaluate model outputs on given recorded data, such as comparing the vehicle trajectory predicted by the model with the reference trajectory in the recorded data to observe whether the prediction error decreases.
Such metrics are beneficial for comparing different models and can also help the R&D team determine whether training is effective. However, the reference trajectory is not the only reasonable driving answer. Under the same road environment, drivers may adopt different but equally reasonable driving trajectories. Therefore, the model output being closer to a certain reference trajectory only indicates an improvement in this specific evaluation metric and cannot independently prove that the vehicle will drive more safely or smoothly.
Real driving is actually a continuous interactive process. After the vehicle takes an action, surrounding traffic participants may react, and the vehicle's own position and speed will also change, resulting in a different environment for the system to face next. A seemingly minor decision deviation may gradually amplify through continuous interactions. In other words, a reduction in open-loop prediction error does not necessarily bring about an improvement in closed-loop driving performance.
Open-loop evaluation mainly examines the model's output under given recorded conditions; closed-loop evaluation further examines whether the system can make rational decisions under continuously changing states after the vehicle takes an action.
For end-to-end autonomous driving models, this point is particularly important. The model not only needs to output reasonable driving actions but also needs to maintain stable driving behavior when its own actions change the subsequent environment. Therefore, it is necessary to combine closed-loop simulation, scenario-based testing, and real-vehicle validation to evaluate the model's performance in continuous interactions and check whether the improvements are stable and whether new failure modes are introduced.
Closed-loop simulation is also affected by factors such as the realism of the simulation environment, the behavior models of traffic participants, and the coverage of test scenarios, and cannot completely replace real-vehicle validation. Different testing methods have their own boundaries, and a single metric or a single test result cannot replace system-level evaluation. For the autonomous driving industry, it is not enough to merely compare single metrics before and after model training; it is also necessary to distinguish which improvements merely make the prediction results closer to the reference trajectory and which improvements truly enhance the vehicle's behavior on the road.
4. What Should Ultimately Be Looked at When Evaluating Autonomous Driving Models?
For autonomous driving models, model scale, data volume, and offline metrics all have value, but they are different observational dimensions in the R&D process. They cannot replace each other, nor can they be directly equated with final driving capabilities. To determine whether a model has become stronger, at least three questions need to be answered.
The first is: what specific problem does the model solve? For example, when facing complex intersections, can the system make more reasonable passing decisions; when facing sudden lane changes by other vehicles, can it adjust the vehicle's behavior in a timely manner and maintain stability in continuous interactions. Capability improvements should correspond to clear driving tasks, rather than merely resting on parameter scale or computing power growth.
The second is: is the model improvement stable? If the model performs better only on some test samples but degrades in other scenarios, the overall capability improvement cannot be simply confirmed. The R&D team needs to observe performance under different road conditions, combinations of traffic participants, and environmental changes, and continuously analyze failure cases.
The last is: can the improvement be validated in closed-loop and real-vehicle tests? Whether a single action is reasonable is only a local performance; it is equally important whether the system can control risks, reduce unnecessary hard braking and trajectory deviations in continuous interactions, and maintain expected behavior under different road conditions.
For scenarios not yet covered by tests, the capability boundaries also need to be clearly defined, rather than directly generalizing the performance in limited tests to all road environments. This is not to say that model scale is unimportant. Larger models, more sufficient data, and higher training investments can all create conditions for capability improvements; it is just that these investments must be matched with effective data, reasonable training objectives, and reliable evaluation methods.
5. Concluding Remarks
Zhijia Qianyan believes that as model scales continue to grow, what the autonomous driving industry truly needs to focus on should not merely be how much the parameters have increased or how much the training data has expanded, but rather what specific driving problems these investments have actually solved and what problems have consistently remained unimproved.
Models can be made larger and larger, but scale itself cannot prove that driving capabilities have improved. Only by mapping model, data, and training investments to specific driving problems and validating the improvement effects through multi-level testing can it be determined whether these investments have truly been translated into driving capabilities.