Abstract:Generalization error (also known as the out-of-sample error) measures how well the hypothesis learned from training data generalizes to previously unseen data. Proving tight generalization error bounds is a central question in statistical learning theory. In this paper, we obtain generalization error bounds for learning general non-convex objectives, which has attracted significant attention in recent years. We develop a new framework, termed Bayes-Stability, for proving algorithm-dependent generalization error bounds. The new framework combines ideas from both the PAC-Bayesian theory and the notion of algorithmic stability. Applying the Bayes-Stability method, we obtain new data-dependent generalization bounds for stochastic gradient Langevin dynamics (SGLD) and several other noisy gradient methods (e.g., with momentum, mini-batch and acceleration, Entropy-SGD). Our result recovers (and is typically tighter than) a recent result in Mou et al. (2018) and improves upon the results in Pensia et al. (2018). Our experiments demonstrate that our data-dependent bounds can distinguish randomly labelled data from normal data, which provides an explanation to the intriguing phenomena observed in Zhang et al. (2017a). We also study the setting where the total loss is the sum of a bounded loss and an additional ℓ_2 regularization term. We obtain new generalization bounds for the continuous Langevin dynamic in this setting by developing a new Log-Sobolev inequality for the parameter distribution at any time. Our new bounds are more desirable when the noisy level of the process is not small, and do not become vacuous even when T tends to infinity.

Generalization Performance of Bipartite Ranking Algorithms with Convex Losses

Adversarial Prediction Games for Multivariate Losses

Generalization Analysis for Ranking Using Integral Operator

Generalization Performance of Regularized Ranking with Multiscale Kernels

Generalization Analysis of Listwise Learning-to-rank Algorithms

Two-Layer Generalization Analysis for Ranking Using Rademacher Average.

Fine-grained analysis of non-parametric estimation for pairwise learning

Rethinking and Reweighting the Univariate Losses for Multi-Label Ranking: Consistency and Generalization

Online Learning with Pairwise Loss Functions

On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning

Generalization Analysis For Ranking

A New Family of Generalization Bounds Using Samplewise Evaluated CMI

Generalization Error Bounds for Iterative Learning Algorithms with Bounded Updates

Top Rank Optimization in Linear Time

Towards Generalization Beyond Pointwise Learning: A Unified Information-theoretic Perspective

Bipartite Ranking Fairness through a Model Agnostic Ordering Adjustment

BiRank: Towards Ranking on Bipartite Graphs

Low-rank matrix recovery with non-quadratic loss: projected gradient method and regularity projection oracle

Learning with Convex Loss and Indefinite Kernels

On loss functions and regret bounds for multi-category classification

Multistage Convex Relaxation Approach to Rank Regularized Minimization Problems Based on Equivalent Mathematical Program with a Generalized Complementarity Constraint