Abstract:The generalization error of a learning algorithm refers to the discrepancy between the loss of a learning algorithm on training data and that on unseen testing data. Various information-theoretic bounds on the generalization error have been derived in the literature, where the mutual information between the training data and the hypothesis (the output of the learning algorithm) plays an important role. Focusing on the individual sample mutual information bound by Bu et al., which itself is a tightened version of the first bound on the topic by Russo et al. and Xu et al., this paper investigates the tightness of these bounds, in terms of the dependence of their convergence rates on the sample size $n$. It has been recognized that these bounds are in general not tight, readily verified for the exemplary quadratic Gaussian mean estimation problem, where the individual sample mutual information bound scales as $O(\sqrt{1/n})$ while the true generalization error scales as $O(1/n)$. The first contribution of this paper is to show that the same bound can in fact be asymptotically tight if an appropriate assumption is made. In particular, we show that the fast rate can be recovered when the assumption is made on the excess risk instead of the loss function, which was usually done in existing literature. A theoretical justification is given for this choice. The second contribution of the paper is a new set of generalization error bounds based on the $(\eta, c)$-central condition, a condition relatively easy to verify and has the property that the mutual information term directly determines the convergence rate of the bound. Several analytical and numerical examples are given to show the effectiveness of these bounds.

Generalized Estimation and Information

Bounds smaller than the Fisher information for generalized linear models

Generalization Bounds via Conditional $f$-Information

The Quasi-Likelihood for Generalized Linear Models Revisited

Inference for High-Dimensional Linear Expectile Regression with De-Biasing Method

Rethinking Mean Square Error: Why Information is a Superior Assessment of Estimators

Fast Rate Information-theoretic Bounds on Generalization Errors

Towards Generalization Beyond Pointwise Learning: A Unified Information-theoretic Perspective

Generalized Centroid Estimators in Bioinformatics

Generalized Information Bottleneck for Gaussian Variables

Estimators of Entropy and Information via Inference in Probabilistic Models

Generalizing Geometric Partition Entropy for the Estimation of Mutual Information in the Presence of Informative Outliers

Optimum estimation via gradients of partition functions and information measures: a statistical-mechanical perspective

On the Tightness of Information-Theoretic Bounds on Generalization Error of Learning Algorithms.

Geometric Variational Inference

Generalized multi-view model: Adaptive density estimation under low-rank constraints

Scaling Limits of the Wasserstein information matrix on Gaussian Mixture Models

How to estimate Fisher information matrices from simulations

Parameter estimation and uncertainty quantification using information geometry

An Information-Theoretic Approach to Universal Feature Selection in High-Dimensional Inference.

Exactly Tight Information-Theoretic Generalization Error Bound for the Quadratic Gaussian Problem