Abstract:With the implementation of the new EU regulation 2022/1426 regarding the type-approval of the automated driving system (ADS) of fully automated vehicles, scenario-based testing has gained significant importance in evaluating the performance and safety of advanced driver assistance systems and automated driving systems. However, the exploration and generation of concrete scenarios from a single logical scenario can often lead to a number of similar or redundant scenarios, which may not contribute to the testing goals.
This paper focuses on the the goal to reduce the scenario set by clustering concrete scenarios from a single logical scenario. By employing clustering techniques, redundant and uninteresting scenarios can be identified and eliminated, resulting in a representative scenario set. This reduction allows for a more focused and efficient testing process, enabling the allocation of resources to the most relevant and critical scenarios. Furthermore, the identified clusters can provide valuable insights into the scenario space, revealing patterns and potential problems with the system's behavior.
What problem does this paper attempt to address?
### Problems Addressed by the Paper
This paper aims to address a critical issue in the testing of Automated Driving Systems (ADS): how to identify and eliminate redundant or unimportant scenarios generated from a single logical scenario, thereby reducing the number of scenarios that need to be tested. Specifically, the paper focuses on the following three main objectives:
1. **Assessing Criticality**: Evaluating criticality within the space of a single logical scenario to identify key areas.
2. **Two Scenario Clustering Methods**:
- **Behavior-Based Clustering**: Classifying scenarios based on the similarity of actor behaviors to identify clusters with similar outcomes.
- **Criticality-Based Clustering**: Clustering scenarios based on the process of criticality changes to group high-risk and low-risk scenarios.
3. **Scenario Set Reduction**: Eliminating redundant and unimportant scenarios through the aforementioned clustering methods to obtain a more concise and manageable set of scenarios.
### Background and Motivation
With the implementation of EU Regulation 2022/1426, scenario-based testing has become crucial in evaluating the performance and safety of Advanced Driver Assistance Systems (ADAS) and Automated Driving Systems. However, specific scenarios generated from a single logical scenario often contain a large number of similar or redundant scenarios that may not contribute to the testing objectives. Therefore, reducing the number of scenario sets and focusing resources on testing the most critical and relevant scenarios has become an urgent problem to solve.
### Main Contributions
1. **Assessing Criticality**: Identifying key areas by evaluating criticality within the scenario space.
2. **Two Scenario Clustering Methods**:
- **Behavior-Based Clustering**: Classifying scenarios based on the similarity of actor behaviors.
- **Criticality-Based Clustering**: Clustering scenarios based on the process of criticality changes.
3. **Scenario Set Reduction**: Eliminating redundant and unimportant scenarios through clustering methods to obtain a more concise set of scenarios.
4. **Reduction in Computation Time**: Compared to grid-based methods, using clustering methods can significantly reduce computation time.
### Experimental Setup and Methods
The paper conducts two different experimental setups, each corresponding to a logical scenario:
1. **Scenario 1**: In a crossroad scenario, including an ego vehicle (E), a pedestrian (P), a second car (C), and a truck (T). The ego vehicle makes a right turn, crossing the paths of the pedestrian and the truck but not the second car. The parameters varied in the experiment include the starting position of the ego vehicle, the waiting time of the pedestrian before crossing, and the speed of the second car.
2. **Scenario 2**: In a T-junction scenario, the ego vehicle makes a right turn while the second car goes straight through the junction. Both vehicles have a synchronization point, with the second car attempting to reach the synchronization point simultaneously with the ego vehicle. The parameters varied in the experiment include the coordinates of the ego vehicle's synchronization point, the starting time of the second car, and the distance the ego vehicle must travel before initializing the second car's path.
### Results and Discussion
1. **Scenario 1**: Through approximately 400 executions of specific scenarios, it was found that one parameter (p1_speed) had minimal impact on the scenario outcome. All criticality metrics exhibited similar patterns within the scenario space, with each metric identifying a key diagonal in the lower-left corner. Using Kernel Principal Component Analysis (Kernel PCA) and Density-Based Spatial Clustering of Applications with Noise (DBSCAN), two main clusters were identified: high criticality and low criticality scenarios. Ultimately, through prototype analysis, 32 representative scenarios were extracted from 15 scenarios defining cluster boundaries.
2. **Scenario 2**: Through approximately 300 executions of specific scenarios, it was found that all three parameters affected the scenario outcome. Scenario 2 exhibited three different outcomes: (1) Vehicle C passes through the junction before the ego vehicle, corresponding to low criticality scenarios; (2) a collision between the two vehicles; (3) the ego vehicle passes through the junction before Vehicle C. Using Dynamic Time Warping (DTW) to measure the distance between ego vehicle trajectories, collisions were identified as a separate cluster, resulting in three main clusters: non-collision cases and intersection collisions.
### Conclusion
By using Bayesian optimization and Gaussian processes to explore the space of logical scenarios, and employing behavior-based and criticality-based clustering methods, the paper successfully identified and reduced redundant and unimportant scenarios, resulting in a more concise and manageable set of scenarios. This not only improved testing efficiency but also...