Abstract:Generalization error (also known as the out-of-sample error) measures how well the hypothesis learned from training data generalizes to previously unseen data. Proving tight generalization error bounds is a central question in statistical learning theory. In this paper, we obtain generalization error bounds for learning general non-convex objectives, which has attracted significant attention in recent years. We develop a new framework, termed Bayes-Stability, for proving algorithm-dependent generalization error bounds. The new framework combines ideas from both the PAC-Bayesian theory and the notion of algorithmic stability. Applying the Bayes-Stability method, we obtain new data-dependent generalization bounds for stochastic gradient Langevin dynamics (SGLD) and several other noisy gradient methods (e.g., with momentum, mini-batch and acceleration, Entropy-SGD). Our result recovers (and is typically tighter than) a recent result in Mou et al. (2018) and improves upon the results in Pensia et al. (2018). Our experiments demonstrate that our data-dependent bounds can distinguish randomly labelled data from normal data, which provides an explanation to the intriguing phenomena observed in Zhang et al. (2017a). We also study the setting where the total loss is the sum of a bounded loss and an additional ℓ_2 regularization term. We obtain new generalization bounds for the continuous Langevin dynamic in this setting by developing a new Log-Sobolev inequality for the parameter distribution at any time. Our new bounds are more desirable when the noisy level of the process is not small, and do not become vacuous even when T tends to infinity.

Localized generalization error and its application to RBFNN training

Localized Generalization Error And Its Application To Rbfnn Training

Localized Generalization Error Model and Its Application to Architecture Selection for Radial Basis Function Neural Network

Radial Basis Function Network Learning Using Localized Generalization Error Bound

Feature Selection Using Localized Generalization Error for Supervised Classification Problems Using RBFNN

Norm-Based Localized Generalization Error Model And Its Derivation For Radial Basis Function Neural Networks

Construction of radial basis function neural networks via a minimization of its localized generalization error

A Novel Learning Objective Function Using Localized Generalization Error Bound For Rbf Network

Localized Generalization Error of Gaussian-based Classifiers and Visualization of Decision Boundaries

The Localized Generalization Error Model for Single Layer Perceptron Neural Network and Sigmoid Support Vector Machine.

Construction engineering decision support enabled by minimization of localized generalization error of RBFNN

Image Classification with the Use of Radial Basis Function Neural Networks and the Minimization of the Localized Generalization Error.

Active learning using localized generalization error of candidate sample as criterion

Feature Weighting for RBFNN Based on Genetic Algorithm and Localized Generalization Error Model.

On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning

Generalization and Estimation Error Bounds for Model-based Neural Networks

A Novel Rbf Neural Network With Fast Training And Accurate Generalization

Localized generalization error model for Multilayer Perceptron Neural Networks

Generalization and Risk Bounds for Recurrent Neural Networks

Heuristic improvement for active learning using localized generalization error as selection criterion

A Novel Radial Basis Function Neural Network with High Generalization Performance for Nonlinear Process Modelling