Abstract:The staggering progress of machine learning in the past decade has been a sight to behold. In retrospect, it is both remarkable and unsettling that these milestones were achievable with little to no rigorous theory to guide experimentation. Despite this fact, practitioners have been able to guide their future experimentation via observations from previous large-scale empirical investigations. However, alluding to Plato's Allegory of the cave, it is likely that the observations which form the field's notion of reality are but shadows representing fragments of that reality. In this work, we propose a theoretical framework which attempts to answer what exists outside of the cave. To the theorist, we provide a framework which is mathematically rigorous and leaves open many interesting ideas for future exploration. To the practitioner, we provide a framework whose results are very intuitive, general, and which will help form principles to guide future investigations. Concretely, we provide a theoretical framework rooted in Bayesian statistics and Shannon's information theory which is general enough to unify the analysis of many phenomena in machine learning. Our framework characterizes the performance of an optimal Bayesian learner, which considers the fundamental limits of information. Throughout this work, we derive very general theoretical results and apply them to derive insights specific to settings ranging from data which is independently and identically distributed under an unknown distribution, to data which is sequential, to data which exhibits hierarchical structure amenable to meta-learning. We conclude with a section dedicated to characterizing the performance of misspecified algorithms. These results are exciting and particularly relevant as we strive to overcome increasingly difficult machine learning challenges in this endlessly complex world.

An Information Theoretic Framework for Distributed Learning Algorithms

On Distributed Learning with Constant Communication Bits

On the Optimal Error Exponent of Type-Based Distributed Hypothesis Testing

Distributed Information Theoretic Clustering

Diffusion Information Theoretic Learning for Distributed Estimation over Network

Learning over inherently distributed data

An Information-theoretic Method for Collaborative Distributed Learning with Limited Communication

Information-Theoretic Foundations for Machine Learning

Learning Theory of Distributed Spectral Algorithms

On Distributed Hypothesis Testing with Constant-Bit Communication Constraints

On the Fundamental Limit of Distributed Learning with Interchangable Constrained Statistics

Distributed Machine Learning with Strategic Network Design: A Game-Theoretic Perspective

Distributed Learning Systems with First-order Methods

Robustness in Wireless Distributed Learning: An Information-Theoretic Analysis

On the Trade-off between Flatness and Optimization in Distributed Learning

Toward Quality of Information Aware Distributed Machine Learning

Optimal Inference for Distributed Detection

Privacy-Preserving Distributed Optimization and Learning

Random Inverse Problems Over Graphs: Decentralized Online Learning

Information Theoretic Limits of Data Shuffling for Distributed Learning

Locally Differentially Private Distributed Online Learning with Guaranteed Optimality