Abstract:Deep neural networks (DNNs) trained with the logistic loss (also known as the cross entropy loss) have made impressive advancements in various binary classification tasks. Despite the considerable success in practice, generalization analysis for binary classification with deep neural networks and the logistic loss remains scarce. The unboundedness of the target function for the logistic loss in binary classification is the main obstacle to deriving satisfactory generalization bounds. In this paper, we aim to fill this gap by developing a novel theoretical analysis and using it to establish tight generalization bounds for training fully connected ReLU DNNs with logistic loss in binary classification. Our generalization analysis is based on an elegant oracle -type inequality which enables us to deal with the boundedness restriction of the target function. Using this oracle -type inequality, we establish generalization bounds for fully connected ReLU DNN classifiers f FNN n trained by empirical logistic risk minimization with respect to i.i.d. samples of size n, which lead to sharp rates of convergence as n -> infinity. In particular, we obtain optimal convergence rates f FNN n (up to some logarithmic factor) only requiring the Holder smoothness of the confor ditional class probability eta of data. Moreover, we consider a compositional assumption that requires eta to be the composition of several vector -valued multivariate functions of which each component function is either a maximum value function or a Holder smooth function only depending on a small number of its input variables. Under this assumption, we can f FNN n (up to some logarithmic factor) which are even derive optimal convergence rates for independent of the input dimension of data. This result explains why in practice DNN classifiers can overcome the curse of dimensionality and perform well in high -dimensional classification problems. Furthermore, we establish dimension -free rates of convergence under other circumstances such as when the decision boundary is piecewise smooth and the input data are bounded away from it. Besides the novel oracle -type inequality, the sharp convergence rates presented in our paper also owe to a tight error bound for approximating the natural logarithm function near zero (where it is unbounded) by ReLU DNNs. In addition, we justify our claims for the optimality of rates by proving corresponding minimax lower bounds. All these results are new in the literature and will deepen our theoretical understanding of classification with deep neural networks.

A Geometric Theory for Binary Classification of Finite Datasets by DNNs with Relu Activations

Hyperbolic Binary Neural Network

Classification with Deep Neural Networks and Logistic Loss

A singular Riemannian Geometry Approach to Deep Neural Networks III. Piecewise Differentiable Layers and Random Walks on $n$-dimensional Classes

The Geometric Structure of Fully-Connected ReLU Layers

Classification of Data Generated by Gaussian Mixture Models Using Deep ReLU Networks

The Real Tropical Geometry of Neural Networks

Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations

Information-Theoretic Generalization Bounds for Deep Neural Networks

Fundamental Limits of Deep Learning-Based Binary Classifiers Trained with Hinge Loss

A Mathematical Principle of Deep Learning: Learn the Geodesic Curve in the Wasserstein Space

Implicit Hypersurface Approximation Capacity in Deep ReLU Networks

How deep learning works --The geometry of deep learning

Adjustable Bounded Rectifiers: Towards Deep Binary Representations

A theoretical framework for deep locally connected ReLU network

BinaryConnect: Training Deep Neural Networks with binary weights during propagations

Topological Expressivity of ReLU Neural Networks

Deep Learning as Ricci Flow

On Training a Neural Network to Explain Binaries

On Geometric Structure of Activation Spaces in Neural Networks