Abstract:With the rapid recent development, deep learning based object detection techniques have been applied to various real-world software systems, especially in safety-critical applications like autonomous driving. However, few studies are conducted to systematically investigate the robustness of state-of-the-art object detection techniques against real-world image corruptions and yet few benchmarks of object detection methods in terms of robustness are publicly available. To bridge this gap, we initiate to create a public benchmark of COCO-C and BDD100K-C, composed of sixteen real-world corruptions according to the real damages in camera sensors and image pipeline. Based on that, we further perform a systematic empirical study and evaluation of twelve representative object detectors covering three different categories of architectures ( i.e ., two-stage, one-stage, transformer architectures) to identify the current challenges and explore future opportunities. Our key findings include (1) the proposed real-world corruptions pose a threat to object detectors, especially for the corruptions involving colour changes, (2) a detector with a high mAP may still be vulnerable to real-world corruptions, (3) if there are potential cross-scenarios applications, the one-stage detectors are recommended, (4) when object detection architectures suffer from real-world corruptions, the effectiveness of existing robustness enhancement methods is limited, and (5) two-stage and one-stage object detection architectures are more likely to miss detect objects compared with transformer-based methods against the proposed corruptions. Our results highlight the need for designing robust object detection methods against real-world corruption and the need for more effective robustness enhancement methods for existing object detectors.

Using Synthetic Corruptions to Measure Robustness to Natural Distribution Shifts

Using the Overlapping Score to Improve Corruption Benchmarks

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

Are Adversarial Robustness and Common Perturbation Robustness Independent Attributes ?

Benchmarking the Robustness of Deep Neural Networks to Common Corruptions in Digital Pathology

Benchmarking the Robustness of Spatial-Temporal Models Against Corruptions

Towards Improving Robustness Against Common Corruptions using Mixture of Class Specific Experts

Benchmarking Object Detection Robustness against Real-World Corruptions

A Survey on the Robustness of Computer Vision Models against Common Corruptions

Assessing Visually-Continuous Corruption Robustness of Neural Networks Relative to Human Performance

Investigating the Corruption Robustness of Image Classifiers with Random Lp-norm Corruptions

Benchmarking Robustness to Text-Guided Corruptions

Benchmarking Robustness of 3D Point Cloud Recognition Against Common Corruptions

Benchmarking Low-Shot Robustness to Natural Distribution Shifts

Robust Validation: Confident Predictions Even When Distributions Shift

Generalizability of Adversarial Robustness Under Distribution Shifts

Dynamic Batch Norm Statistics Update for Natural Robustness

Reassessing the Validity of Spurious Correlations Benchmarks

Utilizing Class Separation Distance for the Evaluation of Corruption Robustness of Machine Learning Classifiers

OODRobustBench: a Benchmark and Large-Scale Analysis of Adversarial Robustness under Distribution Shift

Robustness to Spurious Correlations via Human Annotations