Abstract:In this paper, we study the data-dependent convergence and generalization behavior of gradient methods for neural networks with smooth activation. Our first result is a novel bound on the excess risk of deep networks trained by the logistic loss, via an alogirthmic stability analysis. Compared to previous works, our results improve upon the shortcomings of the well-established Rademacher complexity-based bounds. Importantly, the bounds we derive in this paper are tighter, hold even for neural networks of small width, do not scale unfavorably with width, are algorithm-dependent, and consequently capture the role of initialization on the sample complexity of gradient descent for deep nets. Specialized to noiseless data separable with margin $\gamma$ by neural tangent kernel (NTK) features of a network of width $\Omega(\poly(\log(n)))$, we show the test-error rate to be $e^{O(L)}/{\gamma^2 n}$, where $n$ is the training set size and $L$ denotes the number of hidden layers. This is an improvement in the test loss bound compared to previous works while maintaining the poly-logarithmic width conditions. We further investigate excess risk bounds for deep nets trained with noisy data, establishing that under a polynomial condition on the network width, gradient descent can achieve the optimal excess risk. Finally, we show that a large step-size significantly improves upon the NTK regime's results in classifying the XOR distribution. In particular, we show for a one-hidden-layer neural network of constant width $m$ with quadratic activation and standard Gaussian initialization that mini-batch SGD with linear sample complexity and with a large step-size $\eta=m$ reaches the perfect test accuracy after only $\ceil{\log(d)}$ iterations, where $d$ is the data dimension.

Learning Rates for Classification with Gaussian Kernels

Generalized Convexity-Based Inexact Projection Method for Multiple Kernel Learning

Adaptive Learning Rates for Support Vector Machines Working on Data with Low Intrinsic Dimension

A Learning Algorithm with Gaussian Regularizer for Kernel Neuron

Optimal learning rates for Kernel Conjugate Gradient regression

Online Kernel Learning with a Near Optimal Sparsity Bound

Learning Rates of L^q Coefficient Regularization Learning with Gaussian Kernel

Error Scaling Laws for Kernel Classification under Source and Capacity Conditions

Learning Rates for Kernel-Based Expectile Regression

Optimal Learning Rates for Localized SVMs

Learning with smooth Hinge losses

Subgradient-Based Neural Network for Nonconvex Optimization Problems in Support Vector Machines with Indefinite Kernels

The Optimality of Kernel Classifiers in Sobolev Space

Optimal Gaussian Kernel Parameter Selection for SVM Classifier

How to Make the Gradients Small Privately: Improved Rates for Differentially Private Non-Convex Optimization

Support Vector Classification and Gaussian Kernel with Multiple Widths

Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methods

GEV Regression with Convex Loss Applied to Imbalanced Binary Classification.

Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints.

Pairwise Ranking with Gaussian Kernel

Sharp Analysis for Nonconvex SGD Escaping from Saddle Points