Abstract:The staggering progress of machine learning in the past decade has been a sight to behold. In retrospect, it is both remarkable and unsettling that these milestones were achievable with little to no rigorous theory to guide experimentation. Despite this fact, practitioners have been able to guide their future experimentation via observations from previous large-scale empirical investigations. However, alluding to Plato's Allegory of the cave, it is likely that the observations which form the field's notion of reality are but shadows representing fragments of that reality. In this work, we propose a theoretical framework which attempts to answer what exists outside of the cave. To the theorist, we provide a framework which is mathematically rigorous and leaves open many interesting ideas for future exploration. To the practitioner, we provide a framework whose results are very intuitive, general, and which will help form principles to guide future investigations. Concretely, we provide a theoretical framework rooted in Bayesian statistics and Shannon's information theory which is general enough to unify the analysis of many phenomena in machine learning. Our framework characterizes the performance of an optimal Bayesian learner, which considers the fundamental limits of information. Throughout this work, we derive very general theoretical results and apply them to derive insights specific to settings ranging from data which is independently and identically distributed under an unknown distribution, to data which is sequential, to data which exhibits hierarchical structure amenable to meta-learning. We conclude with a section dedicated to characterizing the performance of misspecified algorithms. These results are exciting and particularly relevant as we strive to overcome increasingly difficult machine learning challenges in this endlessly complex world.

Exploring the Truth and Beauty of Theory Landscapes with Machine Learning

Seeking Truth and Beauty in Flavor Physics with Machine Learning

Perspective: Energy Landscapes for Machine Learning

Deep-Learning the Landscape

Machine Learning and Serving of Discrete Field Theories -- when Artificial Intelligence Meets the Discrete Universe

Evaluating Loss Landscapes from a Topology Perspective

Machine learning in physics and geometry

Building Healthy Lagrangian Theories With Machine Learning

Towards Novel Insights in Lattice Field Theory with Explainable Machine Learning

From the String Landscape to the Mathematical Landscape: a Machine-Learning Outlook

Physics-Inspired Interpretability Of Machine Learning Models

Machine learning and serving of discrete field theories

Information-Theoretic Foundations for Machine Learning

Presence and Absence of Barren Plateaus in Tensor-Network Based Machine Learning

Machine Learning Topological States

Discovering the Building Blocks of Atomic Systems using Machine Learning

Approach the Gell-Mann-Okubo Formula with Machine Learning

Landscaping Linear Mode Connectivity

Machine learning of discrete field theories with guaranteed convergence and uncertainty quantification

Machine learning and optimization-based approaches to duality in statistical physics

Exploring QCD matter in extreme conditions with Machine Learning