Abstract:Deep neural networks are at the heart of safety-critical applications such as autonomous driving. Distributional shift is a typical problem in predictive modeling, when the feature distribution of inputs and outputs varies between the training and test stages. When used on data different from the training distribution, neural networks provide little or no performance guarantees on such out-of-distribution (OOD) inputs. Monitoring distributional shift can help assess reliability of neural network predictions with the purpose of predicting potential safety-critical contexts. With our research, we evaluate state of the art OOD detection methods on autonomous driving camera data, while also demonstrating the influence of OOD data on the prediction reliability of neural networks. We evaluate three different OOD detection methods: As a baseline method we employ a variational autoencoder (VAE) trained on the similar data as the perception network (depth estimation) and use a reconstruction error based out of distribution measure. As a second approach, we choose to evaluate a method termed Likelihood Regret, which has been shown to be an efficient likelihood based OOD measure for VAEs. As a third approach, we evaluate another recently introduced method based on generative modelling termed SSD, which uses self-supervised representation learning followed by a distance based detection in the feature space, to calculate the outlier score. We compare all 3 methods and evaluate them concurrently with the error of an depth estimation network. Results show that while the reconstruction error based OOD metric is not able to differentiate between in and out of distribution data across all scenarios, the likelihood regret based OOD metric as well as the SSD outlier score perform fairly well in OOD detection. Their metrics are also highly correlated with perception error, rendering them promising candidates for an autonomous driving system reliability monitor.

Natural Attribute-based Shift Detection

Spatial Anomaly Detection in Hyperspectral Imaging Using Optical Neural Networks

Automatic dataset shift identification to support root cause analysis of AI performance drift

Anomaly Detection under Distribution Shift

Natural Synthetic Anomalies for Self-Supervised Anomaly Detection and Localization

FOOD: Fast Out-Of-Distribution Detector

Prediction Accuracy & Reliability: Classification and Object Localization under Distribution Shift

Monitoring Perception Reliability in Autonomous Driving: Distributional Shift Detection for Estimating the Impact of Input Data on Prediction Accuracy

Natural Adversarial Objects

Detecting Out-of-distribution Objects Using Neuron Activation Patterns

Detection and Retrieval of Out-of-Distribution Objects in Semantic Segmentation

ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms

Mahalanobis-Aware Training for Out-of-Distribution Detection

Detecting Out-of-Distribution Examples with In-distribution Examples and Gram Matrices

Expecting The Unexpected: Towards Broad Out-Of-Distribution Detection

Exploring Endogenous Shift for Cross-domain Detection: A Large-scale Benchmark and Perturbation Suppression Network

Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts

DC4L: Distribution Shift Recovery via Data-Driven Control for Deep Learning Models

Toward Improving Robustness of Object Detectors Against Domain Shift

Non-Linear Outlier Synthesis for Out-of-Distribution Detection

Unexplored Faces of Robustness and Out-of-Distribution: Covariate Shifts in Environment and Sensor Domains