Abstract:The generalization error of a learning algorithm refers to the discrepancy between the loss of a learning algorithm on training data and that on unseen testing data. Various information-theoretic bounds on the generalization error have been derived in the literature, where the mutual information between the training data and the hypothesis (the output of the learning algorithm) plays an important role. Focusing on the individual sample mutual information bound by Bu et al., which itself is a tightened version of the first bound on the topic by Russo et al. and Xu et al., this paper investigates the tightness of these bounds, in terms of the dependence of their convergence rates on the sample size $n$. It has been recognized that these bounds are in general not tight, readily verified for the exemplary quadratic Gaussian mean estimation problem, where the individual sample mutual information bound scales as $O(\sqrt{1/n})$ while the true generalization error scales as $O(1/n)$. The first contribution of this paper is to show that the same bound can in fact be asymptotically tight if an appropriate assumption is made. In particular, we show that the fast rate can be recovered when the assumption is made on the excess risk instead of the loss function, which was usually done in existing literature. A theoretical justification is given for this choice. The second contribution of the paper is a new set of generalization error bounds based on the $(\eta, c)$-central condition, a condition relatively easy to verify and has the property that the mutual information term directly determines the convergence rate of the bound. Several analytical and numerical examples are given to show the effectiveness of these bounds.

Lower Bounds on the Expected Excess Risk Using Mutual Information

Lower and Upper Bounds on the Generalization of Stochastic Exponentially Concave Optimization.

Information Theoretic Lower Bounds for Information Theoretic Upper Bounds

Excess Risk Bounds for Exponentially Concave Losses

On the Tightness of Information-Theoretic Bounds on Generalization Error of Learning Algorithms.

Minimax Excess Risk of First-Order Methods for Statistical Learning with Data-Dependent Oracles

Towards Sharper Risk Bounds for Minimax Problems

Minimum Excess Risk in Bayesian Learning

Binary Excess Risk for Smooth Convex Surrogates.

On Excess Risk Convergence Rates of Neural Network Classifiers

Fast Rate Information-theoretic Bounds on Generalization Errors

Information Lower Bounds for Robust Mean Estimation

Exponential inequalities under the sub-linear expectations with applications to laws of the iterated logarithm

Stability and Sharper Risk Bounds with Convergence Rate $O(1/n^2)$

A Universal Growth Rate for Learning with Smooth Surrogate Losses

Generalization Error Bounds for Noisy, Iterative Algorithms via Maximal Leakage

Optimal Excess Risk Bounds for Empirical Risk Minimization on $p$-Norm Linear Regression

High-probability minimax lower bounds

Learning Adversarially Robust Representations via Worst-Case Mutual Information Maximization

Error Bounds of Supervised Classification from Information-Theoretic Perspective

Learning Rates for Kernel-Based Expectile Regression