Abstract:We investigate the in-distribution generalization of machine learning algorithms. We depart from traditional complexity-based approaches by analyzing information-theoretic bounds that quantify the dependence between a learning algorithm and the training data. We consider two categories of generalization guarantees: 1) Guarantees in expectation: These bounds measure performance in the average case. Here, the dependence between the algorithm and the data is often captured by information measures. While these measures offer an intuitive interpretation, they overlook the geometry of the algorithm's hypothesis class. Here, we introduce bounds using the Wasserstein distance to incorporate geometry, and a structured, systematic method to derive bounds capturing the dependence between the algorithm and an individual datum, and between the algorithm and subsets of the training data. 2) PAC-Bayesian guarantees: These bounds measure the performance level with high probability. Here, the dependence between the algorithm and the data is often measured by the relative entropy. We establish connections between the Seeger--Langford and Catoni's bounds, revealing that the former is optimized by the Gibbs posterior. We introduce novel, tighter bounds for various types of loss functions. To achieve this, we introduce a new technique to optimize parameters in probabilistic statements. To study the limitations of these approaches, we present a counter-example where most of the information-theoretic bounds fail while traditional approaches do not. Finally, we explore the relationship between privacy and generalization. We show that algorithms with a bounded maximal leakage generalize. For discrete data, we derive new bounds for differentially private algorithms that guarantee generalization even with a constant privacy parameter, which is in contrast to previous bounds in the literature.

The generalized universal law of generalization

The universal law of generalization holds for naturalistic stimuli.

The Universal Law of Generalization Holds for Naturalistic Stimuli

Generalization, similarity, and Bayesian inference

Evidence for the size principle in semantic and perceptual domains

Unifying Principles of Generalization: Past, Present, and Future

On the Complexity of Bayesian Generalization

On the generality of the laws of learning.

An Information-Theoretic Approach to Generalization Theory

Universal properties of spaces of generalized functions

A unified framework for information-theoretic generalization bounds

Is generalization decay a fundamental law of psychology?

Is Complexity an Illusion?

1 Generalization in Classical Statistical Learning Theory

An Optimal Transport View on Generalization.

A mathematical theory of relational generalization in transitive inference

Learning Whenever Learning is Possible: Universal Learning under General Stochastic Processes

Representations and generalization in artificial and brain neural networks

Generalizability of experimental studies

Structure of universal formulas

Algorithmic Randomness and Probabilistic Laws