Abstract:In spite of the dominant performances of deep neural networks, recent works have shown that they are poorly calibrated, resulting in over-confident predictions. Miscalibration can be exacerbated by overfitting due to the minimization of the cross-entropy during training, as it promotes the predicted softmax probabilities to match the one-hot label assignments. This yields a pre-softmax activation of the correct class that is significantly larger than the remaining activations. Recent evidence from the literature suggests that loss functions that embed implicit or explicit maximization of the entropy of predictions yield state-of-the-art calibration performances. We provide a unifying constrained-optimization perspective of current state-of-the-art calibration losses. Specifically, these losses could be viewed as approximations of a linear penalty (or a Lagrangian) imposing equality constraints on logit distances. This points to an important limitation of such underlying equality constraints, whose ensuing gradients constantly push towards a non-informative solution, which might prevent from reaching the best compromise between the discriminative performance and calibration of the model during gradient-based optimization. Following our observations, we propose a simple and flexible generalization based on inequality constraints, which imposes a controllable margin on logit distances. Comprehensive experiments on a variety of image classification, semantic segmentation and NLP benchmarks demonstrate that our method sets novel state-of-the-art results on these tasks in terms of network calibration, without affecting the discriminative performance. The code is available at <a class="link-external link-https" href="https://github.com/by-liu/MbLS" rel="external noopener nofollow">this https URL</a> .

Advancing neural network calibration: The role of gradient decay in large-margin Softmax optimization

The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration

Understanding the Generalization Benefits of Late Learning Rate Decay

An automatic learning rate decay strategy for stochastic gradient descent optimization methods in neural networks

The Calibration Generalization Gap

Why Does Sharpness-Aware Minimization Generalize Better Than SGD?

Gradient Descent Maximizes the Margin of Homogeneous Neural Networks.

Stochastic Gradient Descent Introduces an Effective Landscape-Dependent Regularization Favoring Flat Solutions

A Large-Scale Study of Probabilistic Calibration in Neural Network Regression

How Does Learning Rate Decay Help Modern Neural Networks?

Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate

Probabilistic Calibration by Design for Neural Network Regression

Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapult

Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methods

Selective Learning: Towards Robust Calibration with Dynamic Regularization

On the Limitations of Temperature Scaling for Distributions with Overlaps

Adaptive temperature scaling for Robust calibration of deep neural networks

Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks

An Effective Dynamic Gradient Calibration Method for Continual Learning

Towards Unbiased Calibration using Meta-Regularization

Optimizing Calibration by Gaining Aware of Prediction Correctness