ACAV: A Framework for Automatic Causality Analysis in Autonomous Vehicle Accident Recordings

Huijia Sun,Christopher M. Poskitt,Yang Sun,Jun Sun,Yuqi Chen

DOI: https://doi.org/10.1145/3597503.3639175

2024-01-13

Abstract:The rapid progress of autonomous vehicles~(AVs) has brought the prospect of a driverless future closer than ever. Recent fatalities, however, have emphasized the importance of safety validation through large-scale testing. Multiple approaches achieve this fully automatically using high-fidelity simulators, i.e., by generating diverse driving scenarios and evaluating autonomous driving systems~(ADSs) against different test oracles. While effective at finding violations, these approaches do not identify the decisions and actions that \emph{caused} them -- information that is critical for improving the safety of ADSs. To address this challenge, we propose ACAV, an automated framework designed to conduct causality analysis for AV accident recordings in two stages. First, we apply feature extraction schemas based on the messages exchanged between ADS modules, and use a weighted voting method to discard frames of the recording unrelated to the accident. Second, we use safety specifications to identify safety-critical frames and deduce causal events by applying CAT -- our causal analysis tool -- to a station-time graph. We evaluate ACAV on the Apollo ADS, finding that it can identify five distinct types of causal events in 93.64% of 110 accident recordings generated by an AV testing engine. We further evaluated ACAV on 1206 accident recordings collected from versions of Apollo injected with specific faults, finding that it can correctly identify causal events in 96.44% of the accidents triggered by prediction errors, and 85.73% of the accidents triggered by planning errors.

Software Engineering

What problem does this paper attempt to address?

The problem addressed in this paper is how to automatically perform causal analysis in accident records of autonomous vehicles (AV) to identify the specific decisions and actions that led to the accidents, in order to improve the safety of AVs.

The ACAV framework is a two-stage automated tool designed to tackle this problem. Firstly, it simplifies the accident records by extracting features from the messages exchanged between ADS modules and using a weighted voting method to filter out irrelevant parts. Then, it identifies safety-critical keyframes using safety specifications and analyzes the timeline graph using a causal analysis tool called CAT to determine the causal events of the accidents. In the evaluation, ACAV identified 5 types of causal events in 110 accident records and correctly identified most accidents caused by prediction errors and planning errors in 1206 accident records of a specific fault-injected Apollo version.