When an autonomous driving vehicle encounters an issue on the road, the system records relevant data. This data is then transmitted back to the cloud, where it undergoes filtering, annotation, and training, ultimately improving driving performance through model updates.
This process sounds like it forms a complete data closed loop.
But has the problem truly been identified once the data returns? Does completing sample annotation mean the model has really learned? Will vehicle performance on real roads definitely improve after the model is trained?
In some autonomous driving R&D processes, data collection, transmission, and training have reached scale, yet gaps remain in the chain from problem discovery to capability validation.
Data volume continues to grow, and models are constantly iterating, but certain problems may still recur repeatedly.
Behind this phenomenon lies a question worth asking: Is the data closed loop actually driving systemic capability improvements, or is it merely completing the collection and flow of data?
01. Does More Transmitted Data Mean More Value?
For autonomous driving systems, data collection is not just about recording the vehicle's driving process.
Depending on the vehicle model and sensor configuration, vehicles can collect camera images, LiDAR point clouds, millimeter-wave radar information, as well as data on vehicle speed, steering angle, braking status, and system operating state.
This data helps R&D personnel reconstruct the road environment at the time, the status of traffic participants, and the perception and decision-making processes of the autonomous driving system.
However, the volume of data generated daily by vehicles is enormous. Not all data needs to be uploaded, nor does all data hold equal value. What truly impacts model capabilities are often the difficult scenarios that expose system weaknesses.
For example, when a vehicle passes through a construction zone where original lane lines intersect with temporary markings and surrounding vehicles constantly cut in, the system might exhibit unreasonable deceleration, hesitation in lane changing, or abnormal trajectory planning.
These scenarios are more worthy of attention than straight-line driving on normal roads. Therefore, autonomous driving companies need to establish a data filtering mechanism.
Common signals for data filtering include driver takeovers, system anomalies, unstable perception results, deviations between predicted trajectories and actual motion, and abnormal planned trajectories.
Companies can trigger data recording or uploading based on these signals, then use cloud-based scenario mining and manual analysis to filter out truly valuable problem samples.
However, there is an easily overlooked issue here: trigger signals do not necessarily indicate a real problem occurred.
A driver takeover might be due to a system judgment error, the driver being unfamiliar with the system's driving style, or a proactive takeover based on personal driving preferences.
Conversely, the absence of a takeover does not guarantee that the system's behavior was reasonable. In some cases, while the system shows no obvious errors, it may already exhibit unnecessary deceleration, excessive conservatism, or unnatural decision-making.
If companies only filter data based on the number of takeovers, hard brakes, or anomaly triggers, they may miss issues that do not trigger obvious alerts but affect the actual driving experience.
Furthermore, the data collection mechanism itself suffers from coverage bias.
If the system is more likely to record anomalies that have already been identified but fails to detect risks not yet perceived by the system, the dataset may remain concentrated on known issues while lacking coverage of unknown scenarios.
Therefore, the first step of the data closed loop should not simply be transmitting as much data as possible, but establishing mechanisms capable of discovering, distinguishing, and filtering problems.
Otherwise, as data scale expands, the effective samples that truly drive system improvement may not increase proportionally.
02. Why Might Models Still Fail to Learn Even After Data Annotation?
After data filtering, companies must also process the samples to clarify what scenarios are included, the status of traffic participants, and exactly what issues arose in the autonomous driving system.
This involves automatic annotation, manual annotation, and problem analysis.
Taking construction zones as an example, R&D personnel need to identify cones, barriers, vehicles, and lane lines, while also assessing the currently passable area based on road structure and analyzing why the vehicle did not choose an appropriate driving path.
If only obstacle recognition and bounding box annotation are completed without further analysis of the system's abnormal behavior, the data may enter the training pipeline but fail to resolve the original problem.
This is because anomalies in autonomous driving systems do not always stem from perception errors. For instance, in a construction zone, the perception system may have correctly identified cones and temporary lane lines, but if the planning module generates an overly conservative trajectory that fails to fully utilize available space, the vehicle will still exhibit unnecessary deceleration or detours.
If R&D personnel attribute the problem solely to insufficient perception accuracy and add more annotated samples of related obstacles, the model may not achieve the expected improvement.
For traditional modular autonomous driving systems, R&D personnel can troubleshoot the source of the problem by examining outputs from perception, prediction, and planning modules step by step.
However, in end-to-end autonomous driving systems, perception, prediction, and planning capabilities may be handled jointly by a unified model, making intermediate processes difficult to directly observe and interpret.
In such cases, problem analysis cannot rely solely on the output of one module. It requires comprehensive analysis combining model inputs/outputs, accessible intermediate features, training samples, and actual vehicle behavior. When necessary, controlled experiments and ablation studies should be used to isolate issues.
Additionally, consistency in data annotation is crucial.
In similar construction scenarios, if different annotators have significant discrepancies in judging temporary lane lines, passable areas, or reasonable driving trajectories, it introduces label noise into the training data, reduces consistency, and affects the model's learning effectiveness for those scenarios.
For some end-to-end autonomous driving models using supervised learning, the training process relies on human driving behaviors, planned trajectories, or other forms of supervision signals.
If these signals themselves are biased, the model may learn unreasonable driving behaviors.
Therefore, the value of data annotation lies not just in tagging objects in a scene, but in helping R&D personnel understand why the system failed and how training data should reflect reasonable driving behavior.
If the root cause attribution is inaccurate, subsequent training may merely repeat the use of data that fails to truly solve the problem.
03. Why Do Problems Recur Despite Continuous Model Training?
Once data is filtered and annotated, it enters the model training phase.
Many companies continuously increase training data volume, expand coverage of difficult scenarios, and improve system performance through model iteration.
However, increased data volume does not necessarily mean the model gains stronger generalization capabilities.
A key reason is that the model may repeatedly encounter similar scenarios during training without truly learning to handle new variations.
For example, some companies collect vast amounts of construction road data featuring different roads, weather conditions, and traffic participants.
But if these samples primarily come from the same batch of roads with similar construction layouts, the model may perform well in familiar scenarios but still exhibit unstable judgments when encountering construction zones with different road structures and traffic conditions.
This relates to the representativeness of training data, as well as the division methods for training, validation, and test sets.
In continuous video data, adjacent frames often share high similarity.
If adjacent frames from the same driving video are randomly assigned to training and test sets, the test set may contain many near-duplicate scenes the model has already seen during training.
This data leakage makes test results appear better than they are, failing to reflect the model's true performance on unseen scenarios. Therefore, dataset splitting cannot rely solely on simple random sampling; it must be designed according to evaluation goals.
To validate the model's generalization to new roads, isolation by geographic location is recommended. To avoid distributing similar samples from continuous videos across sets, splitting by driving segments or collection sequences should be considered.
This does not mean data from similar scenarios lacks value. On the contrary, adding samples for similar problems helps improve the model's handling of those scenarios.
But without reasonable isolation between training and testing, it is difficult for companies to determine whether the model is learning transferable driving skills or simply performing well within a familiar data distribution.
Another issue is that model training does not automatically solve all problems.
When new data differs in distribution from existing data, model capabilities may fluctuate. When training samples are overly concentrated on specific scenarios, the model's handling of other scenarios may also suffer. Even if performance improves on a specific issue in the training set, it does not directly prove the vehicle has stable coping abilities on real roads.
Therefore, model training requires attention not only to data scale and training metrics but also to differences between samples, the rationality of dataset splits, and model performance in unseen scenarios.
04. Does Model Update Truly Complete the Closed Loop?
After data processing and model training, the next step is validating whether improvements are effective.
However, improved offline test performance does not guarantee better performance in real-world traffic environments.
A common validation method is historical data replay. R&D personnel input previously collected road data into the model to observe its perception, prediction, and planning results in identical scenarios.
Note that historical data replay is not inherently equivalent to open-loop testing.
If the system input strictly follows historical records and is unaffected by the new model's output, it is open-loop replay. If the simulation environment can regenerate subsequent scenarios based on the ego vehicle's new decisions and motion states, it forms a closed-loop re-simulation.
The distinction lies in whether the vehicle's behavior influences subsequent environmental changes.
In open-loop replay, the trajectories of historical traffic participants are fixed. Even if the new model generates different driving actions, other vehicles continue moving according to the original records. This method is suitable for comparing model outputs on the same historical data but cannot fully reflect the subsequent traffic changes caused by different driving decisions.
Closed-loop simulation allows interaction between the vehicle and the simulated environment.
For example, when an autonomous vehicle prepares to change lanes, other vehicles in the simulation can adjust their speed or position based on the ego vehicle's actions, creating a new traffic state.
However, the effectiveness of closed-loop simulation depends on the accuracy of environment modeling and traffic participant behavior models. If other vehicles follow preset trajectories without responding to the autonomous vehicle's behavior, the simulation's fidelity to real-world traffic interactions remains limited.
Thus, simulation tests help companies discover issues and compare solutions but cannot completely replace real-road validation.
In actual R&D, companies must also combine regression testing, closed-field testing, and road testing to check if model updates introduce new issues or resolve existing ones.
For instance, a model update might improve detour capabilities in construction zones but make the vehicle overly aggressive when meeting on narrow roads.
Validating only the performance of new scenarios without conducting regression tests on existing ones may solve one problem while introducing another.
Model deployment and subsequent vehicle version updates must also be included in the complete validation process.
Only when improvements are validated in both test environments and real roads, without significantly degrading performance in other scenarios, can the iteration be considered practically effective.
If, after a model update, companies fail to continue observing vehicle performance, collecting new issues, and conducting the next round of analysis, the data closed loop effectively remains stuck at the training and deployment stages.
05. The True Competition in Data Closed Loops Goes Beyond Data Scale
From data collection, scenario filtering, and annotation analysis to model training, simulation validation, and real-vehicle testing, the data closed loop is not a single independent step but a continuously running R&D workflow.
Data scale is certainly important.
More real-world data expands scenario coverage and provides richer samples for model training. But data scale is not synonymous with autonomous driving capability.
What truly determines data value is whether companies can identify meaningful problems from massive datasets, accurately pinpoint root causes, transform problems into effective training samples, and confirm through validation that the model has indeed improved.
This explains why companies with similar data collection scales may still exhibit differences in final autonomous driving capabilities.
Some companies can translate anomalous road behaviors into clear R&D tasks, continuously solving problems through data analysis, model training, and validation testing;
Others may accumulate vast amounts of driving data but suffer from gaps in problem filtering, annotation attribution, and validation, leading to increasing data volumes without noticeable practical improvements.
Data scale determines how many real-world scenarios a company can access, while data closed-loop efficiency determines how much capability improvement can be derived from those scenarios.
For the autonomous driving industry, competition in the data closed loop is shifting from data collection capabilities to the overall efficiency of problem discovery, root cause analysis, model iteration, and engineering validation.
If these links are not connected, even the largest data scale may result in nothing more than an ever-expanding data warehouse.
What the data closed loop truly needs to close is not the transmission path from edge to cloud, but the complete chain from real-world problems to engineering improvements.