Abstract:Deep neural networks are easily attacked by imperceptible perturbation. Presently, adversarial training (AT) is the most effective method to enhance the robustness of the model against adversarial examples. However, because adversarial training solved a min-max value problem, in comparison with natural training, the robustness and generalization are contradictory, i.e., the robustness improvement of the model will decrease the generalization of the model. To address this issue, in this paper, a new concept, namely confidence threshold (CT), is introduced and the reducing of the confidence threshold, known as confidence threshold reduction (CTR), is proven to improve both the generalization and robustness of the model. Specifically, to reduce the CT for natural training (i.e., for natural training with CTR), we propose a mask-guided divergence loss function (MDL) consisting of a cross-entropy loss term and an orthogonal term. The empirical and theoretical analysis demonstrates that the MDL loss improves the robustness and generalization of the model simultaneously for natural training. However, the model robustness improvement of natural training with CTR is not comparable to that of adversarial training. Therefore, for adversarial training, we propose a standard deviation loss function (STD), which minimizes the difference in the probabilities of the wrong categories, to reduce the CT by being integrated into the loss function of adversarial training. The empirical and theoretical analysis demonstrates that the STD based loss function can further improve the robustness of the adversarially trained model on basis of guaranteeing the changeless or slight improvement of the natural accuracy.

Empirical study on tangent loss function for classification with deep neural networks

On Loss Functions for Deep Neural Networks in Classification

Taming the Cross Entropy Loss

Can Cross Entropy Loss Be Robust to Label Noise?

Generalized Correntropy Induced Loss Function for Deep Learning.

Misclassification-guided loss under the weighted cross-entropy loss framework

Long-tailed Visual Recognition via Gaussian Clouded Logit Adjustment

MPCE: A Maximum Probability Based Cross Entropy Loss Function for Neural Network Classification

Improving the Robustness and Generalization of Deep Neural Network with Confidence Threshold Reduction

Rethinking Feature Distribution for Loss Functions in Image Classification

Statistical Loss and Analysis for Deep Learning in Hyperspectral Image Classification

Alternate Loss Functions for Classification and Robust Regression Can Improve the Accuracy of Artificial Neural Networks

Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels

Generalized Correntropy Based Deep Learning in Presence of Non-Gaussian Noises.

Robust Loss Functions under Label Noise for Deep Neural Networks

Learning Towards The Largest Margins.

Research on online marketing effects based on multi-model fusion and artificial intelligence algorithms

Next Generation Loss Function for Image Classification

Feature-Balanced Loss for Long-Tailed Visual Recognition

"Lossless" Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approach