ICSFuzz: Collision Detector Bug Discovery in Autonomous Driving Simulators

Weiwei Fu,Heqing Huang,Yifan Zhang,Ke Zhang,Jin Huang,Wei-Bin Lee,Jianping Wang

2024-08-11

Abstract:With the increasing adoption of autonomous vehicles, ensuring the reliability of autonomous driving systems (ADSs) deployed on autonomous vehicles has become a significant concern. Driving simulators have emerged as crucial platforms for testing autonomous driving systems, offering realistic, dynamic, and configurable environments. However, existing simulation-based ADS testers have largely overlooked the reliability of the simulators, potentially leading to overlooked violation scenarios and subsequent safety security risks during real-world deployment. In our investigations, we identified that collision detectors in simulators could fail to detect and report collisions in certain collision scenarios, referred to as ignored collision scenarios. This paper aims to systematically discover ignored collision scenarios to improve the reliability of autonomous driving simulators. To this end, we present ICSFuzz, a black-box fuzzing approach to discover ignored collision scenarios efficiently. Drawing upon the fact that the ignored collision scenarios are a sub-type of collision scenarios, our approach starts with the determined collision scenarios. Following the guidance provided by empirically studied factors contributing to collisions, we selectively mutate arbitrary collision scenarios in a step-wise manner toward the ignored collision scenarios and effectively discover them. We compare ICSFuzz with DriveFuzz, a state-of-the-art simulation-based ADS testing method, by replacing its oracle with our ignored-collision-aware oracle. The evaluation demonstrates that ICSFuzz outperforms DriveFuzz by finding 10-20x more ignored collision scenarios with a 20-70x speedup. All the discovered ignored collisions have been confirmed by developers with one CVE ID assigned.

Cryptography and Security

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to solve the problem that the collision detector in the autonomous driving simulator fails to correctly detect and report collisions, namely the so - called "Ignored Collision Scenarios (ICSs)". Specifically:

1. **Background and Motivation**:
- With the popularization of autonomous vehicles, it is crucial to ensure the reliability of the Autonomous Driving Systems (ADS) deployed on these vehicles. The driving simulator, as an important platform for testing ADS, provides a realistic, dynamic, and configurable environment.
- However, most of the existing simulation - based ADS testing tools ignore the reliability of the simulator itself, which may lead to some collision scenarios being ignored, thus bringing potential safety hazards in actual deployment.

2. **Problem Definition**:
- The paper points out that the existing collision detectors may have false positives or false negatives. Especially in certain specific collision scenarios, the collision detector may not be able to correctly detect collisions, resulting in "ignored collision scenarios".
- If these ignored collision scenarios occur on actual roads, they may lead to serious traffic accidents.

3. **Research Objectives**:
- The goal of the paper is to systematically discover these ignored collision scenarios to improve the reliability of the autonomous driving simulator.
- To this end, the authors propose ICSFuzz, a black - box fuzz - testing - based method that can efficiently discover ignored collision scenarios.

4. **Methodology**:
- ICSFuzz starts from known collision scenarios and gradually mutates to generate new test scenarios in order to find those scenarios that will lead to ignored collisions.
- This method utilizes the research on real - world collision data, determines the key factors affecting collisions, and generates new test inputs by selectively mutating these factors.

5. **Contributions**:
- This is the first work to study the reliability of the autonomous driving simulator and discover ignored collision scenarios.
- Experiments have proven that ICSFuzz has a significant performance improvement over the existing state - of - the - art simulator testing methods (such as DriveFuzz) in discovering ignored collision scenarios. It can discover more types of ignored collision scenarios and has been confirmed by developers and assigned a CVE ID.

In summary, this paper aims to systematically discover and solve the vulnerabilities of the collision detector in the autonomous driving simulator by proposing a new method, ICSFuzz, thereby improving the reliability and safety of the simulator.