Abstract:Generalization error (also known as the out-of-sample error) measures how well the hypothesis learned from training data generalizes to previously unseen data. Proving tight generalization error bounds is a central question in statistical learning theory. In this paper, we obtain generalization error bounds for learning general non-convex objectives, which has attracted significant attention in recent years. We develop a new framework, termed Bayes-Stability, for proving algorithm-dependent generalization error bounds. The new framework combines ideas from both the PAC-Bayesian theory and the notion of algorithmic stability. Applying the Bayes-Stability method, we obtain new data-dependent generalization bounds for stochastic gradient Langevin dynamics (SGLD) and several other noisy gradient methods (e.g., with momentum, mini-batch and acceleration, Entropy-SGD). Our result recovers (and is typically tighter than) a recent result in Mou et al. (2018) and improves upon the results in Pensia et al. (2018). Our experiments demonstrate that our data-dependent bounds can distinguish randomly labelled data from normal data, which provides an explanation to the intriguing phenomena observed in Zhang et al. (2017a). We also study the setting where the total loss is the sum of a bounded loss and an additional ℓ_2 regularization term. We obtain new generalization bounds for the continuous Langevin dynamic in this setting by developing a new Log-Sobolev inequality for the parameter distribution at any time. Our new bounds are more desirable when the noisy level of the process is not small, and do not become vacuous even when T tends to infinity.

Rademacher Complexity and the Generalization Error of Residual Networks

On Rademacher Complexity-based Generalization Bounds for Deep Learning

Generalization and Estimation Error Bounds for Model-based Neural Networks

Theoretical Investigation of Generalization Bound for Residual Networks.

Bridging the Gap: Rademacher Complexity in Robust and Standard Generalization

Algorithm-Dependent Generalization Bounds for Overparameterized Deep Residual Networks

Rademacher Complexity for Adversarially Robust Generalization.

Generalization and Risk Bounds for Recurrent Neural Networks

Dropout Training, Data-dependent Regularization, and Generalization Bounds.

Generalization Error Bounds for Deep Neural Networks Trained by SGD

Why ResNet Works? Residuals Generalize

A Priori Estimates of the Population Risk for Residual Networks

Understanding the Generalization Ability of Deep Learning Algorithms: A Kernelized Rényi’s Entropy Perspective

Learning Non-Vacuous Generalization Bounds from Optimization

Understanding the Generalization Ability of Deep Learning Algorithms: A Kernelized Renyi's Entropy Perspective

On the Lipschitz Constant of Deep Networks and Double Descent

Generalization bounds for graph convolutional neural networks via Rademacher complexity

Generalization Ability of Wide Residual Networks

On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning

On Tighter Generalization Bound for Deep Neural Networks: CNNs, ResNets, and Beyond

Generalization Error of Generalized Linear Models in High Dimensions