Abstract:With the growing capabilities of autonomous vehicles, there is a higher demand for sophisticated and pragmatic quality assurance approaches for machine learning-enabled systems in the automotive AI context. The use of simulation-based prototyping platforms provides the possibility for early-stage testing, enabling inexpensive testing and the ability to capture critical corner-case test scenarios. Simulation-based testing properly complements conventional on-road testing. However, due to the large space of test input parameters in these systems, the efficient generation of effective test scenarios leading to the unveiling of failures is a challenge. This paper presents a study on testing pedestrian detection and emergency braking system of the Baidu Apollo autonomous driving platform within the SVL simulator. We propose an evolutionary automated test generation technique that generates failure-revealing scenarios for Apollo in the SVL environment. Our approach models the input space using a generic and flexible data structure and benefits a multi-criteria safety-based heuristic for the objective function targeted for optimization. This paper presents the results of our proposed test generation technique in the 2021 IEEE Autonomous Driving AI Test Challenge. In order to demonstrate the efficiency and effectiveness of our approach, we also report the results from a baseline random generation technique. Our evaluation shows that the proposed evolutionary test case generator is more effective at generating failure-revealing test cases and provides higher diversity between the generated failures than the random baseline.

Improving Failure Detection by Automatically Generating Test Cases Near the Boundaries.

Effective code coverage in compositional systematic dynamic testing

Optimal test case generation for boundary value analysis

Improving Defect Detection Ability of Derived Test Cases Based on Mutated UML Activity Diagrams

Adaptive Random Test Case Generation for Combinatorial Testing

Guided test generation for coverage criteria

Test-Data Generation Guided by Static Defect Detection

Search-Based Software Test Data Generation for Path Coverage Based on a Feedback-Directed Mechanism

Improving Effectiveness of Automated Software Testing in the Absence of Specifications

BATON: Symphony of Random Testing and Concolic Testing Through Machine Learning and Taint Analysis

Adaptive Random Testing by Exclusion Through Test Profile

Identification of Failure Regions for Programs With Numeric Inputs

Hybrid Test Data Generation

Failure-detection Capability Analysis of Implementing Parallelism in Adaptive Random Testing Algorithms

An Empirical Study on Automated Test Generation Tools for Java: Effectiveness and Challenges

Tool-assisted Unit Test Selection Based on Operational Violations

Can Search-Based Testing with Pareto Optimization Effectively Cover Failure-Revealing Test Inputs?

Efficient and Effective Generation of Test Cases for Pedestrian Detection -- Search-based Software Testing of Baidu Apollo in SVL

Tool-assisted Unit-Test Generation and Selection Based on Operational Abstractions

An Approach to Generate Software Test Data for a Specific Path Automatically with Genetic Algorithm

New Random Testing-based Fault Localization Approach