Abstract:Autonomous vehicle (AV) systems must be comprehensively tested and evaluated before they can be deployed. High-fidelity simulators such as CARLA or LGSVL allow this to be done safely in very realistic and highly customizable environments. Existing testing approaches, however, fail to test simulated AVs systematically, as they focus on specific scenarios and oracles (e.g., lane following scenario with the “no collision” requirement) and lack any coverage criteria measures. In this paper, we propose <inline-formula><tex-math notation="LaTeX">$\mathtt {AVUnit}$</tex-math><alternatives><mml:math><mml:mi mathvariant="monospace">AVUnit</mml:mi></mml:math><inline-graphic xlink:href="zhou-ieq1-3254142.gif"/></alternatives></inline-formula>, a framework for systematically testing AV systems against customizable correctness specifications. Designed modularly to support different simulators, <inline-formula><tex-math notation="LaTeX">$\mathtt {AVUnit}$</tex-math><alternatives><mml:math><mml:mi mathvariant="monospace">AVUnit</mml:mi></mml:math><inline-graphic xlink:href="zhou-ieq2-3254142.gif"/></alternatives></inline-formula> consists of two new languages for specifying dynamic properties of scenes (e.g., changing pedestrian behaviour after waypoints) and fine-grained assertions about the AV's journey. <inline-formula><tex-math notation="LaTeX">$\mathtt {AVUnit}$</tex-math><alternatives><mml:math><mml:mi mathvariant="monospace">AVUnit</mml:mi></mml:math><inline-graphic xlink:href="zhou-ieq3-3254142.gif"/></alternatives></inline-formula> further supports multiple fuzzing algorithms that automatically search for test cases that violate these assertions, using robustness and coverage measures as fitness metrics. We evaluated the implementation of <inline-formula><tex-math notation="LaTeX">$\mathtt {AVUnit}$</tex-math><alternatives><mml:math><mml:mi mathvariant="monospace">AVUnit</mml:mi></mml:math><inline-graphic xlink:href="zhou-ieq4-3254142.gif"/></alternatives></inline-formula> for the LGSVL+Apollo simulation environment, finding 19 kinds of issues in Apollo, which indicate that the open-source Apollo does not perform well in complex intersections and lane-changing related scenarios.

Validation on Safety of the Intended Functionality of Automated Vehicles: Concept Development

Runtime unknown unsafe scenarios identification for SOTIF of autonomous vehicles

A Survey on an Emerging Safety Challenge for Autonomous Vehicles: Safety of the Intended Functionality

Automated Vehicle-Involved Traffic Flow Studies: A Survey of Assumptions, Models, Speculations, and Perspectives

Formal Certification Methods for Automated Vehicle Safety Assessment

A taxonomy of validation strategies to ensure the safe operation of highly automated vehicles

Procedure for the Safety Assessment of an Autonomous Vehicle Using Real-World Scenarios

On Quantification for SOTIF Validation of Automated Driving Systems

Genetic Algorithm-Based SOTIF Scenario Construction for Complex Traffic Flow

Autonomous Vehicles Testing Considering Utility-Based Operable Tasks

Scenario-Based Safety Assessment Framework for Automated Vehicles

Efficient falsification approach for autonomous vehicle validation using a parameter optimisation technique based on reinforcement learning

A Quantitative Approach of Generating Challenging Testing Scenarios Based on Functional Safety Standard

Reliability assessment of autonomous vehicles based on the safety control structure

Integrating Formal Verification and Simulation-based Assertion Checking in a Corroborative V&V Process

scenoRITA: Generating Less-Redundant, Safety-Critical and Motion Sickness-Inducing Scenarios for Autonomous Vehicles

The Implementation of Safety Acceptance Evaluation for Highly Automated Vehicles

Evolving Testing Scenario Generation Method and Intelligence Evaluation Framework for Automated Vehicles

Simulation-based Safety Assurance for an AVP System incorporating Learning-Enabled Components

Specification-Based Autonomous Driving System Testing

Few-Shot Scenario Testing for Autonomous Vehicles Based on Neighborhood Coverage and Similarity