Abstract:Deep neural networks (DNNs) trained with the logistic loss (also known as the cross entropy loss) have made impressive advancements in various binary classification tasks. Despite the considerable success in practice, generalization analysis for binary classification with deep neural networks and the logistic loss remains scarce. The unboundedness of the target function for the logistic loss in binary classification is the main obstacle to deriving satisfactory generalization bounds. In this paper, we aim to fill this gap by developing a novel theoretical analysis and using it to establish tight generalization bounds for training fully connected ReLU DNNs with logistic loss in binary classification. Our generalization analysis is based on an elegant oracle -type inequality which enables us to deal with the boundedness restriction of the target function. Using this oracle -type inequality, we establish generalization bounds for fully connected ReLU DNN classifiers f FNN n trained by empirical logistic risk minimization with respect to i.i.d. samples of size n, which lead to sharp rates of convergence as n -> infinity. In particular, we obtain optimal convergence rates f FNN n (up to some logarithmic factor) only requiring the Holder smoothness of the confor ditional class probability eta of data. Moreover, we consider a compositional assumption that requires eta to be the composition of several vector -valued multivariate functions of which each component function is either a maximum value function or a Holder smooth function only depending on a small number of its input variables. Under this assumption, we can f FNN n (up to some logarithmic factor) which are even derive optimal convergence rates for independent of the input dimension of data. This result explains why in practice DNN classifiers can overcome the curse of dimensionality and perform well in high -dimensional classification problems. Furthermore, we establish dimension -free rates of convergence under other circumstances such as when the decision boundary is piecewise smooth and the input data are bounded away from it. Besides the novel oracle -type inequality, the sharp convergence rates presented in our paper also owe to a tight error bound for approximating the natural logarithm function near zero (where it is unbounded) by ReLU DNNs. In addition, we justify our claims for the optimality of rates by proving corresponding minimax lower bounds. All these results are new in the literature and will deepen our theoretical understanding of classification with deep neural networks.

An efficient, provably exact, practical algorithm for the 0-1 loss linear classification problem

Exact Algorithms for 0-1 Integer Programs with Linear Equality Constraints

A nonlinear kernel SVM classifier via L 0 / 1 soft-margin loss with classification performance

Classification with Deep Neural Networks and Logistic Loss

Classification Via Minimum Incremental Coding Length

Linear Separation via Optimism

Loss-Aversively Fair Classification

A Novel Loss Function-based Support Vector Machine for Binary Classification

An Exact Algorithm for 0-1 Polynomial Knapsack Problems

Characterizing the Optimal 0-1 Loss for Multi-class Classification with a Test-time Attacker

Methods and Models for Interpretable Linear Classification

iNALM: An inexact Newton Augmented Lagrangian Method for Zero-One Composite Optimization

Efficient Low-Rank Semidefinite Programming With Robust Loss Functions

On the Optimality of the Simple Bayesian Classifier under Zero-One Loss

LBL: Logarithmic Barrier Loss Function for One-class Classification

Optimal-margin evolutionary classifier

Refined Risk Bounds for Unbounded Losses via Transductive Priors

A New Smooth Approximation to the Zero One Loss with a Probabilistic Interpretation

On the Rates of Convergence from Surrogate Risk Minimizers to the Bayes Optimal Classifier.

Provably Accurate and Scalable Linear Classifiers in Hyperbolic Spaces

Towards exploiting linear regression for multi-class/multi-label classification: an empirical analysis