Abstract:Reliable usage of object detectors require them to be calibrated -- a crucial problem that requires careful attention. Recent approaches towards this involve (1) designing new loss functions to obtain calibrated detectors by training them from scratch, and (2) post-hoc Temperature Scaling (TS) that learns to scale the likelihood of a trained detector to output calibrated predictions. These approaches are then evaluated based on a combination of Detection Expected Calibration Error (D-ECE) and Average Precision. In this work, via extensive analysis and insights, we highlight that these recent evaluation frameworks, evaluation metrics, and the use of TS have notable drawbacks leading to incorrect conclusions. As a step towards fixing these issues, we propose a principled evaluation framework to jointly measure calibration and accuracy of object detectors. We also tailor efficient and easy-to-use post-hoc calibration approaches such as Platt Scaling and Isotonic Regression specifically for object detection task. Contrary to the common notion, our experiments show that once designed and evaluated properly, post-hoc calibrators, which are extremely cheap to build and use, are much more powerful and effective than the recent train-time calibration methods. To illustrate, D-DETR with our post-hoc Isotonic Regression calibrator outperforms the recent train-time state-of-the-art calibration method Cal-DETR by more than 7 D-ECE on the COCO dataset. Additionally, we propose improved versions of the recently proposed Localization-aware ECE and show the efficacy of our method on these metrics as well. Code is available at: <a class="link-external link-https" href="https://github.com/fiveai/detection_calibration" rel="external noopener nofollow">this https URL</a>.

Are Sampling Heuristics Necessary in Object Detectors

Is Heuristic Sampling Necessary in Training Deep Object Detectors?

Are Sampling Heuristics Necessary in Object Detectors?

Revisiting Foreground-Background Imbalance in Object Detectors

Is Sampling Heuristics Necessary in Training Deep Object Detectors?

Crowdsourcing Detection of Sampling Biases in Image Datasets

Training Region-Based Object Detectors with Online Hard Example Mining

Sampling Techniques for Large-Scale Object Detection From Sparsely Annotated Objects

Foreground-Background Imbalance Problem in Deep Object Detectors: A Review

Boosting Semi-Supervised 3D Object Detection with Semi-Sampling

False Positive Sampling-based Data Augmentation for Enhanced 3D Object Detection Accuracy

Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection

S4OD: Semi-Supervised learning for Single-Stage Object Detection

Residual Objectness for Imbalance Reduction

Improving Training Instance Quality in Aerial Image Object Detection With a Sampling-Balance-Based Multistage Network

Better Sampling, towards Better End-to-end Small Object Detection

HPS-Det: Dynamic Sample Assignment with Hyper-Parameter Search for Object Detection

Balanced One-Stage Object Detection by Enhancing the Effect of Positive Samples

On Calibration of Object Detectors: Pitfalls, Evaluation and Baselines

Learning from Noisy Anchors for One-Stage Object Detection.

Calibrated Teacher for Sparsely Annotated Object Detection