Prediction Accuracy & Reliability: Classification and Object Localization under Distribution Shift

Fabian Diet,Moussa Kassem Sbeyti,Michelle Karg
2024-09-05
Abstract:Natural distribution shift causes a deterioration in the perception performance of convolutional neural networks (CNNs). This comprehensive analysis for real-world traffic data addresses: 1) investigating the effect of natural distribution shift and weather augmentations on both detection quality and confidence estimation, 2) evaluating model performance for both classification and object localization, and 3) benchmarking two common uncertainty quantification methods - Ensembles and different variants of Monte-Carlo (MC) Dropout - under natural and close-to-natural distribution shift. For this purpose, a novel dataset has been curated from publicly available autonomous driving datasets. The in-distribution (ID) data is based on cutouts of a single object, for which both class and bounding box annotations are available. The six distribution-shift datasets cover adverse weather scenarios, simulated rain and fog, corner cases, and out-of-distribution data. A granular analysis of CNNs under distribution shift allows to quantize the impact of different types of shifts on both, task performance and confidence estimation: ConvNeXt-Tiny is more robust than EfficientNet-B0; heavy rain degrades classification stronger than localization, contrary to heavy fog; integrating MC-Dropout into selected layers only has the potential to enhance task performance and confidence estimation, whereby the identification of these layers depends on the type of distribution shift and the considered task.
Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the performance degradation of convolutional neural networks (CNNs) in autonomous driving perception tasks under natural distribution shifts (such as bad weather conditions, low visibility, rare objects or unfamiliar traffic environments, etc.). Specifically, the paper aims to: 1. **Study the influence of natural distribution shift and weather augmentation on detection quality and confidence estimation**: By analyzing the influence of different types of natural distribution shift (such as rain, fog, extreme situations, etc.) on classification and object localization tasks, and how these influences change the model's confidence estimation. 2. **Evaluate the performance of the model in classification and object localization tasks**: By comparing the performance of different network architectures (such as ResNet - 50, EfficientNet - B0, ConvNeXt - Tiny) under different distribution shifts, evaluate the robustness and accuracy of these models. 3. **Benchmark two common uncertainty quantification methods**: Namely, the ensemble method (Ensembles) and different variants of Monte Carlo Dropout (MC - Dropout), to evaluate the performance of these methods under natural and near - natural distribution shifts. To achieve the above goals, the author created a new dataset AD - Cifar - 7, which contains images extracted from multiple publicly available autonomous driving datasets and covers multiple types of natural distribution shifts. Through a detailed analysis of these datasets, the paper provides in - depth insights into the influence of different distribution shift types on task performance and confidence estimation.