Abstract:Deep neural networks (DNNs) trained with the logistic loss (also known as the cross entropy loss) have made impressive advancements in various binary classification tasks. Despite the considerable success in practice, generalization analysis for binary classification with deep neural networks and the logistic loss remains scarce. The unboundedness of the target function for the logistic loss in binary classification is the main obstacle to deriving satisfactory generalization bounds. In this paper, we aim to fill this gap by developing a novel theoretical analysis and using it to establish tight generalization bounds for training fully connected ReLU DNNs with logistic loss in binary classification. Our generalization analysis is based on an elegant oracle -type inequality which enables us to deal with the boundedness restriction of the target function. Using this oracle -type inequality, we establish generalization bounds for fully connected ReLU DNN classifiers f FNN n trained by empirical logistic risk minimization with respect to i.i.d. samples of size n, which lead to sharp rates of convergence as n -> infinity. In particular, we obtain optimal convergence rates f FNN n (up to some logarithmic factor) only requiring the Holder smoothness of the confor ditional class probability eta of data. Moreover, we consider a compositional assumption that requires eta to be the composition of several vector -valued multivariate functions of which each component function is either a maximum value function or a Holder smooth function only depending on a small number of its input variables. Under this assumption, we can f FNN n (up to some logarithmic factor) which are even derive optimal convergence rates for independent of the input dimension of data. This result explains why in practice DNN classifiers can overcome the curse of dimensionality and perform well in high -dimensional classification problems. Furthermore, we establish dimension -free rates of convergence under other circumstances such as when the decision boundary is piecewise smooth and the input data are bounded away from it. Besides the novel oracle -type inequality, the sharp convergence rates presented in our paper also owe to a tight error bound for approximating the natural logarithm function near zero (where it is unbounded) by ReLU DNNs. In addition, we justify our claims for the optimality of rates by proving corresponding minimax lower bounds. All these results are new in the literature and will deepen our theoretical understanding of classification with deep neural networks.

$f$-Divergence Based Classification: Beyond the Use of Cross-Entropy

Mutual Information Estimation via $f$-Divergence and Data Derangements

Learning Fair Classifiers via Min-Max F-divergence Regularization

Deep Divergence Learning

A Novel Text Classification Algorithm Based on Naïve Bayes and KL-Divergence

Divergence Measure of Belief Function and Its Application in Data Fusion

Deep Divergence-Based Approach to Clustering

Generalized Divergence-based Decision Making Method with an Application to Pattern Classification

Generalized Cauchy-Schwarz Divergence and Its Deep Learning Applications

On the KL-Divergence-based Robust Satisficing Model

Robust Semi-supervised Learning via $f$-Divergence and $α$-Rényi Divergence

On $f$-Divergence Principled Domain Adaptation: An Improved Framework

Classification with Deep Neural Networks and Logistic Loss

A bag-to-class divergence approach to multiple-instance learning

Assessing prediction error of nonparametric regression and classification under Bregman divergence

Generalization of Cross-Entropy Loss Function for Image Classification

Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through $f$-divergence Minimization

Optimal Posteriors for Chi-squared Divergence based PAC-Bayesian Bounds and Comparison with KL-divergence based Optimal Posteriors and Cross-Validation Procedure

The Kullback-Leibler Divergence Used in Machine Learning Algorithms for Health Care Applications and Hypertension Prediction: A Literature Review

Correlations of Cross-Entropy Loss in Machine Learning

Divergence Regulated Encoder Network for Joint Dimensionality Reduction and Classification