Abstract:The staggering progress of machine learning in the past decade has been a sight to behold. In retrospect, it is both remarkable and unsettling that these milestones were achievable with little to no rigorous theory to guide experimentation. Despite this fact, practitioners have been able to guide their future experimentation via observations from previous large-scale empirical investigations. However, alluding to Plato's Allegory of the cave, it is likely that the observations which form the field's notion of reality are but shadows representing fragments of that reality. In this work, we propose a theoretical framework which attempts to answer what exists outside of the cave. To the theorist, we provide a framework which is mathematically rigorous and leaves open many interesting ideas for future exploration. To the practitioner, we provide a framework whose results are very intuitive, general, and which will help form principles to guide future investigations. Concretely, we provide a theoretical framework rooted in Bayesian statistics and Shannon's information theory which is general enough to unify the analysis of many phenomena in machine learning. Our framework characterizes the performance of an optimal Bayesian learner, which considers the fundamental limits of information. Throughout this work, we derive very general theoretical results and apply them to derive insights specific to settings ranging from data which is independently and identically distributed under an unknown distribution, to data which is sequential, to data which exhibits hierarchical structure amenable to meta-learning. We conclude with a section dedicated to characterizing the performance of misspecified algorithms. These results are exciting and particularly relevant as we strive to overcome increasingly difficult machine learning challenges in this endlessly complex world.

Outline and application of statistical learning theory

Generalization Bounds for Stochastic Gradient Langevin Dynamics: A Unified View Via Information Leakage Analysis

Fairness, Accuracy, and Unreliable Data

Applying statistical learning theory to deep learning

A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models

Statistical Learning Theory: Models, Concepts, and Results

INTRODUCTION TO STATISTICAL LEARNING THEORY AND SUPPORT VECTOR MACHINES

Application for Machine Learning Methods in Financial Risk Management

Statistical Learning with Group Invariance: Problem, Method and Consistency

An Information-Theoretic Analysis for Transfer Learning: Error Bounds and Applications

On the Generalization for Transfer Learning: An Information-Theoretic Analysis

1 Generalization in Classical Statistical Learning Theory

Statistical Robustness of Empirical Risks in Machine Learning

The Generalization Error of Machine Learning Algorithms

Information-Theoretic Foundations for Machine Learning

Statistical Learning Theory For Ranking

Information-theoretic Analysis for Transfer Learning.

Distributional Robustness Bounds Generalization Errors

Broad Minimax Probability Learning System and its Application in Regression Modeling

The Landscape of Deep Learning Algorithms