Deep Learning Safety Concerns in Automated Driving Perception

Stephanie Abrecht,Alexander Hirsch,Shervin Raafatnia,Matthias Woehrle

2024-07-12

Abstract:Recent advances in the field of deep learning and impressive performance of deep neural networks (DNNs) for perception have resulted in an increased demand for their use in automated driving (AD) systems. The safety of such systems is of utmost importance and thus requires to consider the unique properties of DNNs. In order to achieve safety of AD systems with DNN-based perception components in a systematic and comprehensive approach, so-called safety concerns have been introduced as a suitable structuring element. On the one hand, the concept of safety concerns is -- by design -- well aligned to existing standards relevant for safety of AD systems such as ISO 21448 (SOTIF). On the other hand, it has already inspired several academic publications and upcoming standards on AI safety such as ISO PAS 8800. While the concept of safety concerns has been previously introduced, this paper extends and refines it, leveraging feedback from various domain and safety experts in the field. In particular, this paper introduces an additional categorization for a better understanding as well as enabling cross-functional teams to jointly address the concerns.

Machine Learning,Computer Vision and Pattern Recognition,Systems and Control

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to solve the security problems of the deep - learning perception module in the autonomous driving system (AD). Specifically, it focuses on how to systematically identify and classify the safety concerns of deep neural networks (DNN) in the autonomous driving perception tasks to ensure the security of these systems.

#### Background and Motivation of the Paper

With the progress of deep - learning technology, deep neural networks (DNN) have performed excellently in perception, prediction and planning tasks, and are thus widely used in autonomous driving systems (AD). However, in such safety - critical systems, it is crucial to understand the impact of DNN on the overall system safety. In particular, when DNN has insufficient functions, it may lead to insufficient output, which in turn may cause dangerous behaviors of the vehicle. For example, if the traffic sign detection DNN fails to recognize a stop sign due to an adversarial patch in the scene, the vehicle may not stop, thus causing danger.

#### Concept of Safety Problems

To ensure the safety of the autonomous driving system, the paper introduces the concept of "safety problems" as the basis for systematic and comprehensive analysis. A safety problem refers to the root cause that may lead to insufficient DNN functions. Once triggered, these insufficient functions will lead to insufficient output and may ultimately cause dangerous behaviors of the system. Safety problems are consistent with the causal model in ISO 21448 (SOTIF), which describes how trigger conditions activate insufficient functions of system elements, which in turn lead to insufficient output and dangerous behaviors.

#### Main Contributions

The main contributions of the paper include:

1. **Expand and Refine Safety Problems**: Based on previous literature [2], through discussions with domain experts, a more comprehensive and detailed list of safety problems is proposed.
2. **Classify Safety Problems**: Classify safety problems into four categories according to their sources:
- **Open - world context**: Involves changes and uncertainties in the operating environment of autonomous vehicles.
- **Data and data set preparation**: Involves the quality and distribution of data required for training and evaluating DNN.
- **DNN characteristics**: Involves the characteristics of DNN itself, such as over - fitting, generalization ability, etc.
- **Analysis and evaluation**: Involves the performance evaluation methods of DNN within the operational design domain.

#### Specific Problem Examples

Take stop sign recognition as an example. If the DNN fails to correctly recognize a stop sign that has never been seen before (insufficient output), this may be due to the fact that the DNN has not been trained on similar signs (safety problem). This situation may cause the vehicle to fail to stop in time, thus resulting in dangerous behavior.

In conclusion, this paper provides a structured framework for future research by systematizing and classifying safety problems, helping researchers and engineers better understand and solve the security problems of the deep - learning perception module in the autonomous driving system.