Abstract:Deep neural networks (DNNs) trained with the logistic loss (also known as the cross entropy loss) have made impressive advancements in various binary classification tasks. Despite the considerable success in practice, generalization analysis for binary classification with deep neural networks and the logistic loss remains scarce. The unboundedness of the target function for the logistic loss in binary classification is the main obstacle to deriving satisfactory generalization bounds. In this paper, we aim to fill this gap by developing a novel theoretical analysis and using it to establish tight generalization bounds for training fully connected ReLU DNNs with logistic loss in binary classification. Our generalization analysis is based on an elegant oracle -type inequality which enables us to deal with the boundedness restriction of the target function. Using this oracle -type inequality, we establish generalization bounds for fully connected ReLU DNN classifiers f FNN n trained by empirical logistic risk minimization with respect to i.i.d. samples of size n, which lead to sharp rates of convergence as n -> infinity. In particular, we obtain optimal convergence rates f FNN n (up to some logarithmic factor) only requiring the Holder smoothness of the confor ditional class probability eta of data. Moreover, we consider a compositional assumption that requires eta to be the composition of several vector -valued multivariate functions of which each component function is either a maximum value function or a Holder smooth function only depending on a small number of its input variables. Under this assumption, we can f FNN n (up to some logarithmic factor) which are even derive optimal convergence rates for independent of the input dimension of data. This result explains why in practice DNN classifiers can overcome the curse of dimensionality and perform well in high -dimensional classification problems. Furthermore, we establish dimension -free rates of convergence under other circumstances such as when the decision boundary is piecewise smooth and the input data are bounded away from it. Besides the novel oracle -type inequality, the sharp convergence rates presented in our paper also owe to a tight error bound for approximating the natural logarithm function near zero (where it is unbounded) by ReLU DNNs. In addition, we justify our claims for the optimality of rates by proving corresponding minimax lower bounds. All these results are new in the literature and will deepen our theoretical understanding of classification with deep neural networks.

Learning logistic circuits

Learning Logistic Circuits

Learning Interpretable Differentiable Logic Networks

Deep Differentiable Logic Gate Networks

Learning Algorithms via Neural Logic Networks

Convolutional Differentiable Logic Gate Networks

Accuracy Versus Simplification in an Approximate Logic Neural Model

Categories of Differentiable Polynomial Circuits for Machine Learning

Multinomial Latent Logistic Regression For Image Understanding

Classification with Deep Neural Networks and Logistic Loss

Time Series Classification using the Hidden-Unit Logistic Model

Logical Activation Functions: Logit-space equivalents of Probabilistic Boolean Operators

DeepGate: Learning Neural Representations of Logic Gates

CorrLog: Correlated Logistic Models for Joint Prediction of Multiple Labels.

Density-based logistic regression

Cellular automata, many-valued logic, and deep neural networks

Learning Signal Temporal Logic through Neural Network for Interpretable Classification

On Learning with LAD

Modeling the EdNet Dataset with Logistic Regression

Log-Linear RNNs: Towards Recurrent Neural Networks with Flexible Prior Knowledge

Sparse Logistic Regression Learns All Discrete Pairwise Graphical Models