Abstract:Safety and performance are key enablers for autonomous driving: on the one hand we want our autonomous vehicles (AVs) to be safe, while at the same time their performance (e.g., comfort or progression) is key to adoption. To effectively walk the tight-rope between safety and performance, AVs need to be risk-averse, but not entirely risk-avoidant. To facilitate safe-yet-performant driving, in this paper, we develop a task-aware risk estimator that assesses the risk a perception failure poses to the AV's motion plan. If the failure has no bearing on the safety of the AV's motion plan, then regardless of how egregious the perception failure is, our task-aware risk estimator considers the failure to have a low risk; on the other hand, if a seemingly benign perception failure severely impacts the motion plan, then our estimator considers it to have a high risk. In this paper, we propose a task-aware risk estimator to decide whether a safety maneuver needs to be triggered. To estimate the task-aware risk, first, we leverage the perception failure - detected by a perception monitor - to synthesize an alternative plausible model for the vehicle's surroundings. The risk due to the perception failure is then formalized as the "relative" risk to the AV's motion plan between the perceived and the alternative plausible scenario. We employ a statistical tool called copula, which models tail dependencies between distributions, to estimate this risk. The theoretical properties of the copula allow us to compute probably approximately correct (PAC) estimates of the risk. We evaluate our task-aware risk estimator using NuPlan and compare it with established baselines, showing that the proposed risk estimator achieves the best F1-score (doubling the score of the best baseline) and exhibits a good balance between recall and precision, i.e., a good balance of safety and performance.

What problem does this paper attempt to address?

### Problems Addressed by the Paper This paper aims to address the issue of risk assessment caused by failures in the perception systems of autonomous vehicles (AVs) on vehicle motion planning. Specifically, the authors propose a task-aware risk estimator to evaluate the impact of perception failures on the motion planning of autonomous vehicles and to decide whether safety measures need to be triggered. ### Background and Motivation Despite rapid advancements in robotics and autonomous systems, the perception modules of autonomous vehicles still exhibit various failure modes (such as object misclassification, undetected objects, phantom obstacles, out-of-distribution objects, etc.), which can jeopardize the safety of passengers, other drivers, and pedestrians. Therefore, developing methods to detect these perception failures has become particularly important. However, frequent failure detection that triggers safety measures every time can adversely affect the performance of autonomous vehicles. Hence, the goal of this paper is to develop a task-aware perception monitor that triggers safety measures only when perception failures pose a significant risk to the vehicle's motion planning, thereby achieving safe and efficient driving. ### Main Contributions 1. **Formalization of Relative Scene Risk (RSR)**: The authors propose the concept of relative scene risk to quantify the risk of perception failures on the motion planning of autonomous vehicles. 2. **Real-time Estimation Algorithm for RSR**: Using copula statistical tools, the authors develop an algorithm to estimate RSR and provide probabilistic correctness guarantees. 3. **Experimental Validation**: Through experiments on the NuPlan dataset, the authors demonstrate the effectiveness of the proposed task-aware risk estimator in 100 real-world perception failure scenarios, achieving the highest F1 score and a good balance between precision and recall. It can also predict collisions in advance, allowing sufficient time for mitigation measures. ### Method Overview 1. **Perception Failure Detection and Identification**: First, the perception failure detection and identification module identifies the failure modes of the perception system and isolates the modules responsible for these failures. 2. **Possible Scenario Generation**: Then, using the information from the perception failure modes, possible alternative scenarios are generated that support the real-world scenario. 3. **Task-aware Risk Estimation**: Finally, the task-aware risk estimator evaluates the increased risk of perception failures on the motion planning of autonomous vehicles. By comparing the risks between the perceived scenario and possible scenarios, the copula tool is used to estimate tail dependencies and provide Probably Approximately Correct (PAC) estimates. ### Conclusion This paper proposes a task-aware risk estimation method that can accurately assess the impact of perception failures on the motion planning of autonomous vehicles and decide whether safety measures need to be taken when the perception system fails. Experimental results show that this method achieves a good balance between performance and safety, with high accuracy and reliability.

Task-Aware Risk Estimation of Perception Failures for Autonomous Vehicles

Drive with Quantified Risk: A Probabilistic Approach for Occlusion-Aware Trajectory Planning

A Multimodal Data-Driven Approach for Driving Risk Assessment

Safety risk assessment for autonomous vehicle road testing

Risk-Based Safety Envelopes for Autonomous Vehicles Under Perception Uncertainty

REAL-SAP: Real-time Evidence Aware Liable Safety Assessment for Perception in Autonomous Driving

Risk-Driven Design of Perception Systems

Occlusion-Aware Risk Assessment for Autonomous Driving in Urban Environments

Real-time Risk Estimation for Active Road Safety: Leveraging Waymo AV Sensor Data with Hierarchical Bayesian Extreme Value Models

Risk-Aware Neural Navigation From BEV Input for Interactive Driving

Modeling Drivers' Risk Perception via Attention to Improve Driving Assistance

Risk in Stochastic and Robust Model Predictive Path-Following Control for Vehicular Motion Planning

System-Level Safety Monitoring and Recovery for Perception Failures in Autonomous Vehicles

Perceptual Risk-Aware Adaptive Responsibility Sensitive Safety for Autonomous Driving.

Overcoming the Fear of the Dark: Occlusion-Aware Model-Predictive Planning for Automated Vehicles Using Risk Fields

AV-Occupant Perceived Risk Model for Cut-In Scenarios with Empirical Evaluation

How Do We Fail? Stress Testing Perception in Autonomous Vehicles

Risk-Aware Planning and Assignment for Ground Vehicles using Uncertain Perception from Aerial Vehicles

A Counterfactual Safety Margin Perspective on the Scoring of Autonomous Vehicles' Riskiness

RiskMap: A Unified Driving Context Representation for Autonomous Motion Planning in Urban Driving Environment

Risk Quantification and Prediction of Non-Driving-related Tasks on Driver's Critical Intervention Behavior in Autonomous Driving Scenarios