Hossein Yousefizadeh,Shenghui Gu,Lionel C. Briand,Ali Nasr
Abstract:Autonomous Driving Systems (ADSs) rely on Deep Neural Networks, allowing vehicles to navigate complex, open environments. However, the unpredictability of these scenarios highlights the need for rigorous system-level testing to ensure safety, a task usually performed with a simulator in the loop. Though one important goal of such testing is to detect safety violations, there are many undesirable system behaviors, that may not immediately lead to violations, that testing should also be focusing on, thus detecting more subtle problems and enabling a finer-grained analysis. This paper introduces Cooperative Co-evolutionary MEtamorphic test Generator for Autonomous systems (CoCoMEGA), a novel automated testing framework aimed at advancing system-level safety assessments of ADSs. CoCoMEGA combines Metamorphic Testing (MT) with a search-based approach utilizing Cooperative Co-Evolutionary Algorithms (CCEA) to efficiently generate a diverse set of test cases. CoCoMEGA emphasizes the identification of test scenarios that present undesirable system behavior, that may eventually lead to safety violations, captured by Metamorphic Relations (MRs). When evaluated within the CARLA simulation environment on the Interfuser ADS, CoCoMEGA consistently outperforms baseline methods, demonstrating enhanced effectiveness and efficiency in generating severe, diverse MR violations and achieving broader exploration of the test space. These results underscore CoCoMEGA as a promising, more scalable solution to the inherent challenges in ADS testing with a simulator in the loop. Future research directions may include extending the approach to additional simulation platforms, applying it to other complex systems, and exploring methods for further improving testing efficiency such as surrogate modeling.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: **Develop an effective and efficient system - level testing method for identifying potential unsafe behaviors that may occur in Autonomous Driving Systems (ADSs) under diverse and unpredictable scenarios and conditions**. Specifically, this research aims to ensure that ADSs operate safely and reliably in complex, open environments, especially when the behavior of pedestrians or vehicles is unpredictable.
### Problem Background
1. **Scalability and Coverage**
- System testing in the real world cannot exhaust all possible execution scenarios, but in critical systems (such as ADSs), identifying unsafe scenarios is crucial for system protection, even if the probability of these scenarios occurring is very low.
2. **Oracle Problem**
- It is very difficult to define precise criteria to determine the acceptable behavior of ADSs in complex tasks (such as perception and decision - making). The variability and unpredictability of real - world traffic scenarios make it impossible to enumerate all possible results and define corresponding expectations.
3. **Computational Cost and Efficiency**
- The computational overhead of high - fidelity simulation is a major challenge, which limits system testing in practice and the ability to rapidly iterate system improvements.
### Solution
To solve the above problems, the authors propose **Cooperative Co - evolutionary MEtamorphic test Generator for Autonomous systems (CoCoMEGA)**, an automated testing framework that combines Metamorphic Testing (MT) and search - based methods (using the Cooperative Co - evolutionary Algorithm CCEA). CoCoMEGA aims to automatically generate a comprehensive and diverse set of system test cases to effectively identify potential unsafe behaviors in ADSs.
### Main Contributions
1. **Propose CoCoMEGA**: This is the first method that combines MT with CCEA for effective and efficient system - level testing of ADSs.
2. **Apply to Complex Cases**: Tests were carried out on a high - performance ADS containing DNN modules on the industrial - level simulator CARLA.
3. **Evaluate Effectiveness and Efficiency**: Through large - scale experiments and comparison with baseline methods, the significant advantages of CoCoMEGA in identifying test cases that violate MR were demonstrated, showing its ability to explore diverse search spaces.
### Specific Problem Formulation
Let \( S \) represent the space of all possible driving scenarios, and \( Q \) represent the space of all possible perturbations derived from a set of MRs, all of which share the same output relation \( o_r \). Let \( E_{o_r}(s, q) \) be a function that quantifies the degree of violation of the output relation \( o_r \) between the source scenario \( s\in S \) and the subsequent scenario \( q(s) \) obtained after applying \( q\in Q \).
The problem can be formalized as:
\[ SP=\{(s, q)\in S\times Q\mid E_{o_r}(s, q)>0\} \]
That is, find a diverse set of scenario - perturbation pairs \( SP \) such that the generated pairs \((s, q(s))\) violate \( o_r \).
### Challenges
1. **Efficiency**: Real - driving scenarios are complex, involving many dynamic and static objects, such as vehicles, pedestrians, and environmental conditions. Efficiently exploring this large space to identify scenarios that violate MRs is computationally intensive.
2. **Metamorphic Relationship Definition**: MT requires clearly defined MRs that describe the expected behavior or invariants in different test scenarios. Defining comprehensive and meaningful MRs is challenging, especially in ADSs, where system behavior is affected by multiple factors.
3. **Test Scenario Diversity**: Ensuring the diversity of test scenarios is crucial for reliable testing. A lack of diversity may lead to insufficient system evaluation and reduce the effectiveness of identifying violations.
By combining MT and advanced search techniques, especially using CCEA, CoCoMEGA can ensure testing efficiency while covering a wider range of possible scenarios, thereby more effectively identifying potential unsafe behaviors in ADSs.