Specification-Based Autonomous Driving System Testing
Christopher M. Poskitt,Jun Sun,Yang Sun,Zijiang Yang,Yun Tang,Yuan Zhou,Yuqi Chen,Yang Liu
DOI: https://doi.org/10.1109/TSE.2023.3254142
IF: 7.4
2023-06-01
IEEE Transactions on Software Engineering
Abstract:Autonomous vehicle (AV) systems must be comprehensively tested and evaluated before they can be deployed. High-fidelity simulators such as CARLA or LGSVL allow this to be done safely in very realistic and highly customizable environments. Existing testing approaches, however, fail to test simulated AVs systematically, as they focus on specific scenarios and oracles (e.g., lane following scenario with the “no collision” requirement) and lack any coverage criteria measures. In this paper, we propose <inline-formula><tex-math notation="LaTeX">$\mathtt {AVUnit}$</tex-math><alternatives><mml:math><mml:mi mathvariant="monospace">AVUnit</mml:mi></mml:math><inline-graphic xlink:href="zhou-ieq1-3254142.gif"/></alternatives></inline-formula>, a framework for systematically testing AV systems against customizable correctness specifications. Designed modularly to support different simulators, <inline-formula><tex-math notation="LaTeX">$\mathtt {AVUnit}$</tex-math><alternatives><mml:math><mml:mi mathvariant="monospace">AVUnit</mml:mi></mml:math><inline-graphic xlink:href="zhou-ieq2-3254142.gif"/></alternatives></inline-formula> consists of two new languages for specifying dynamic properties of scenes (e.g., changing pedestrian behaviour after waypoints) and fine-grained assertions about the AV's journey. <inline-formula><tex-math notation="LaTeX">$\mathtt {AVUnit}$</tex-math><alternatives><mml:math><mml:mi mathvariant="monospace">AVUnit</mml:mi></mml:math><inline-graphic xlink:href="zhou-ieq3-3254142.gif"/></alternatives></inline-formula> further supports multiple fuzzing algorithms that automatically search for test cases that violate these assertions, using robustness and coverage measures as fitness metrics. We evaluated the implementation of <inline-formula><tex-math notation="LaTeX">$\mathtt {AVUnit}$</tex-math><alternatives><mml:math><mml:mi mathvariant="monospace">AVUnit</mml:mi></mml:math><inline-graphic xlink:href="zhou-ieq4-3254142.gif"/></alternatives></inline-formula> for the LGSVL+Apollo simulation environment, finding 19 kinds of issues in Apollo, which indicate that the open-source Apollo does not perform well in complex intersections and lane-changing related scenarios.
Engineering,Computer Science