Abstract:Autonomous driving technology has advanced significantly with deep learning, but noise and attacks threaten its real-world deployment. While research has revealed vulnerabilities in individual intelligent tasks, a comprehensive evaluation of these impacts across complete end-to-end systems is still underexplored. To address this void, we thoroughly analyze the robustness of four end-to-end autonomous driving systems against various noise and build the RobustE2E Benchmark, including five traditional adversarial attacks and a newly proposed Module-Wise Attack specifically targeting end-to-end autonomous driving in white-box settings, as well as four major categories of natural corruptions (a total of 17 types, with five severity levels) in black-box settings. Additionally, we extend the robustness evaluation from the open-loop model level to the closed-loop case studies of autonomous driving system level. Our comprehensive evaluation and analysis provide valuable insights into the robustness of end-to-end autonomous driving, which may offer potential guidance for targeted improvements to models. For example, (1) even the most advanced end-to-end models suffer large planning failures under minor perturbations, with perception tasks showing the most substantial decline; (2) among adversarial attacks, our Module-Wise Attack poses the greatest threat to end-to-end autonomous driving models, while PGD-l2 is the weakest, and among four categories of natural corruptions, noise and weather are the most harmful, followed by blur and digital distortion being less severe; (3) the integrated, multitask approach results in significantly higher robustness and reliability compared with the simpler design, highlighting the critical role of collaborative multitask in autonomous driving; and (4) the autonomous driving systems amplify the model's lack of robustness, etc. Our research contributes to developing more resilient autonomous driving models and their deployment in the real world.

Interpreting Adversarial Examples and Robustness for Deep Learning-Based Auto-Driving Systems

Attack As Defense: Characterizing Adversarial Examples Using Robustness.

Adversarial Robustness Analysis of LiDAR-included Models in Autonomous Driving

Interpreting Adversarial Examples in Deep Learning: A Review

Adversarial robustness improvement for deep neural networks

Opportunities and Challenges in Deep Learning Adversarial Robustness: A Survey

Impact of Architectural Modifications on Deep Learning Adversarial Robustness

Evaluating Adversarial Attacks on Driving Safety in Vision-Based Autonomous Vehicles

Multi-adversarial Testing and Retraining for Improving the Robustness of Autonomous Driving Policies

A Unified Game-Theoretic Interpretation of Adversarial Robustness

A Survey of Robustness and Safety of 2D and 3D Deep Learning Models Against Adversarial Attacks

Safe Driving Adversarial Trajectory Can Mislead: Towards More Stealthy Adversarial Attack Against Autonomous Driving Prediction Module

Adversarial Driving: Attacking End-to-End Autonomous Driving

Dynamic Adversarial Attacks on Autonomous Driving Systems

Towards robust autonomous driving systems through adversarial test set generation

Improved Robustness and Safety for Autonomous Vehicle Control with Adversarial Reinforcement Learning

RobustE2E: Exploring the Robustness of End-to-End Autonomous Driving

Adversarial Objects Against LiDAR-Based Autonomous Driving Systems

Optimism in the Face of Adversity: Understanding and Improving Deep Learning Through Adversarial Robustness

Understanding Adversarial Robustness Via Critical Attacking Route.

Module-wise Adaptive Adversarial Training for End-to-end Autonomous Driving