Abstract:In real-world autonomous driving, deep learning models can experience performance degradation due to distributional shifts between the training data and the driving conditions encountered. As is typical in machine learning, it is difficult to acquire a large and potentially representative labeled test set to validate models in preparation for deployment in the wild. In this work, we introduce complementary learning, where we use learned characteristics from different training paradigms to detect model errors. We demonstrate our approach by learning semantic and predictive motion labels in point clouds in a supervised and self-supervised manner and detect and classify model discrepancies subsequently. We perform a large-scale qualitative analysis and present LidarCODA, the first dataset with labeled anomalies in lidar point clouds, for an extensive quantitative analysis.

What problem does this paper attempt to address?

The paper primarily aims to address the performance degradation issues encountered by deep learning models in autonomous driving scenarios, particularly those caused by distribution shifts between training data and actual driving conditions during real-world deployment. Specifically, the paper proposes the following points: 1. **Problem Background**: In real-world applications, especially in the field of autonomous driving, deep learning models may experience performance degradation due to distribution differences between training data and actual driving conditions. Typically, it is challenging to obtain a sufficiently large and representative labeled test set to validate the model's performance before field deployment. 2. **Solution**: To address the above issues, the authors propose a method called "Complementary Learning," which leverages features learned under different training paradigms to detect model errors. This method combines supervised learning and self-supervised learning to process point cloud data and further detect and classify inconsistencies and errors within the model. 3. **Contribution Overview**: - Introduced the concept of complementary learning and its application examples for detecting model failures. - Constructed the first real-world dataset, LidarCODA, containing labeled anomalous point cloud data for quantitative analysis in autonomous driving scenarios. 4. **Method Overview**: - Extract semantic motion labels from point cloud data using both supervised and self-supervised methods. - Use these labels for point-level comparison to detect inconsistencies between models and further perform clustering analysis on potential model failures. - Classify model failures through manual review to better understand challenging scenarios. 5. **Experimental Results**: Through manual analysis of a large number of frames and experiments on a dataset containing anomalous data, the method has been proven effective in detecting errors and anomalies within the model, which is crucial for improving model performance. In summary, the main objective of this paper is to detect performance degradation issues in deep learning models in autonomous driving scenarios through a novel method—complementary learning, and to support this goal by constructing a corresponding dataset.

Complementary Learning for Real-World Model Failure Detection

Learning-on-the-Drive: Self-supervised Adaptation of Visual Offroad Traversability Models

Learning to Simulate Realistic LiDARs

Failure Prediction for Autonomous Driving

Language-Enhanced Latent Representations for Out-of-Distribution Detection in Autonomous Driving

SDAC: A Multimodal Synthetic Dataset for Anomaly and Corner Case Detection in Autonomous Driving

A Unified Generative Framework for Realistic Lidar Simulation in Autonomous Driving Systems

LiRTest: Augmenting LiDAR Point Clouds for Automated Testing of Autonomous Driving Systems

Learning 3D Perception from Others' Predictions

Online Analytic Exemplar-Free Continual Learning with Large Models for Imbalanced Autonomous Driving Task

LiDAR View Synthesis for Robust Vehicle Navigation Without Expert Labels

Learning from Maps: Visual Common Sense for Autonomous Driving

Monitoring Perception Reliability in Autonomous Driving: Distributional Shift Detection for Estimating the Impact of Input Data on Prediction Accuracy

Sit Back and Relax: Learning to Drive Incrementally in All Weather Conditions

Federated Learning with Heterogeneous Data Handling for Robust Vehicular Object Detection

Robust Object Detection for Autonomous Driving Based on Semi-Supervised Learning

Deep Lidar CNN to Understand the Dynamics of Moving Vehicles

LiDAR-as-Camera for End-to-End Driving

Practical Accuracy Evaluation for Deep Learning Systems Via Latent Representation Discrepancy.

Multimodal End-to-End Learning for Autonomous Steering in Adverse Road and Weather Conditions

Empowering Corner Case Detection in Autonomous Vehicles with Multimodal Large Language Models