Abstract:We investigate the in-distribution generalization of machine learning algorithms. We depart from traditional complexity-based approaches by analyzing information-theoretic bounds that quantify the dependence between a learning algorithm and the training data. We consider two categories of generalization guarantees: 1) Guarantees in expectation: These bounds measure performance in the average case. Here, the dependence between the algorithm and the data is often captured by information measures. While these measures offer an intuitive interpretation, they overlook the geometry of the algorithm's hypothesis class. Here, we introduce bounds using the Wasserstein distance to incorporate geometry, and a structured, systematic method to derive bounds capturing the dependence between the algorithm and an individual datum, and between the algorithm and subsets of the training data. 2) PAC-Bayesian guarantees: These bounds measure the performance level with high probability. Here, the dependence between the algorithm and the data is often measured by the relative entropy. We establish connections between the Seeger--Langford and Catoni's bounds, revealing that the former is optimized by the Gibbs posterior. We introduce novel, tighter bounds for various types of loss functions. To achieve this, we introduce a new technique to optimize parameters in probabilistic statements. To study the limitations of these approaches, we present a counter-example where most of the information-theoretic bounds fail while traditional approaches do not. Finally, we explore the relationship between privacy and generalization. We show that algorithms with a bounded maximal leakage generalize. For discrete data, we derive new bounds for differentially private algorithms that guarantee generalization even with a constant privacy parameter, which is in contrast to previous bounds in the literature.

Learning and generalization theories of large committee--machines

The committee machine: Computational to statistical gaps in learning a two-layers neural network

Modeling Generalization in Machine Learning: A Methodological and Computational Study

Studying Large Language Model Generalization with Influence Functions

What Do Learning Dynamics Reveal About Generalization in LLM Reasoning?

Statistical physics and practical training of soft-committee machines

Generalization in Machine Learning via Analytical Learning Theory

An Information-Theoretic Approach to Generalization Theory

Leveraging PAC-Bayes Theory and Gibbs Distributions for Generalization Bounds with Complexity Measures

Prediction, Learning, Uniform Convergence, and Scale-sensitive Dimensions

Generalized Convergence Analysis of Tsetlin Machines: A Probabilistic Approach to Concept Learning

Generalization Analysis for Game-Theoretic Machine Learning

A theory of representation learning gives a deep generalisation of kernel methods

Storage Capacity of the Tilinglike Learning Algorithm

Algebraic geometrical methods for hierarchical learning machines

Generalization on the Unseen, Logic Reasoning and Degree Curriculum

Scaling Laws with Hidden Structure

A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit

Adaptive Circuit Behavior and Generalization in Mechanistic Interpretability

A General Theory for Training Learning Machine

More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize