Abstract:Autonomous vehicle (AV) systems must be comprehensively tested and evaluated before they can be deployed. High-fidelity simulators such as CARLA or LGSVL allow this to be done safely in very realistic and highly customizable environments. Existing testing approaches, however, fail to test simulated AVs systematically, as they focus on specific scenarios and oracles (e.g., lane following scenario with the “no collision” requirement) and lack any coverage criteria measures. In this paper, we propose <inline-formula><tex-math notation="LaTeX">$\mathtt {AVUnit}$</tex-math><alternatives><mml:math><mml:mi mathvariant="monospace">AVUnit</mml:mi></mml:math><inline-graphic xlink:href="zhou-ieq1-3254142.gif"/></alternatives></inline-formula>, a framework for systematically testing AV systems against customizable correctness specifications. Designed modularly to support different simulators, <inline-formula><tex-math notation="LaTeX">$\mathtt {AVUnit}$</tex-math><alternatives><mml:math><mml:mi mathvariant="monospace">AVUnit</mml:mi></mml:math><inline-graphic xlink:href="zhou-ieq2-3254142.gif"/></alternatives></inline-formula> consists of two new languages for specifying dynamic properties of scenes (e.g., changing pedestrian behaviour after waypoints) and fine-grained assertions about the AV's journey. <inline-formula><tex-math notation="LaTeX">$\mathtt {AVUnit}$</tex-math><alternatives><mml:math><mml:mi mathvariant="monospace">AVUnit</mml:mi></mml:math><inline-graphic xlink:href="zhou-ieq3-3254142.gif"/></alternatives></inline-formula> further supports multiple fuzzing algorithms that automatically search for test cases that violate these assertions, using robustness and coverage measures as fitness metrics. We evaluated the implementation of <inline-formula><tex-math notation="LaTeX">$\mathtt {AVUnit}$</tex-math><alternatives><mml:math><mml:mi mathvariant="monospace">AVUnit</mml:mi></mml:math><inline-graphic xlink:href="zhou-ieq4-3254142.gif"/></alternatives></inline-formula> for the LGSVL+Apollo simulation environment, finding 19 kinds of issues in Apollo, which indicate that the open-source Apollo does not perform well in complex intersections and lane-changing related scenarios.

Specification-Based Autonomous Driving System Testing

Rigorous Simulation-based Testing for Autonomous Driving Systems -- Targeting the Achilles' Heel of Four Open Autopilots

Autonomous Vehicles Testing Considering Utility-Based Operable Tasks

LawBreaker: An Approach for Specifying Traffic Laws and Fuzzing Autonomous Vehicles

Simulation-based Safety Assurance for an AVP System incorporating Learning-Enabled Components

Pre-Deployment Testing of Low Speed, Urban Road Autonomous Driving in a Simulated Environment

An Indoor Rapid Testing Platform for Autonomous Vehicles Using Vehicle-in-the-Loop Simulation

Closed-Loop Simulation Test Method for Autonomous Vehicles Based on Adversarial Scenarios

Runtime unknown unsafe scenarios identification for SOTIF of autonomous vehicles

Parameter-Based Testing and Debugging of Autonomous Driving Systems

Multimodal Large Language Model Driven Scenario Testing for Autonomous Vehicles

Mixed Test Environment-based Vehicle-in-the-loop Validation - A New Testing Approach for Autonomous Vehicles.

Addressing the IEEE AV Test Challenge with Scenic and VerifAI

A Novel Traffic Simulation Framework for Testing Autonomous Vehicles Using SUMO and CARLA

LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving

A Language for Autonomous Vehicles Testing Oracles

Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World

PAFOT: A Position-Based Approach for Finding Optimal Tests of Autonomous Vehicles

Interactive Critical Scenario Generation for Autonomous Vehicles Testing Based on In-depth Crash Data Using Reinforcement Learning

Perception-Guided Fuzzing for Simulated Scenario-Based Testing of Autonomous Driving Systems

Vehicle-in-the-Loop Intelligent Connected Vehicle Simulation System Based on Vehicle-Road-Cloud Collaboration