Abstract:The staggering progress of machine learning in the past decade has been a sight to behold. In retrospect, it is both remarkable and unsettling that these milestones were achievable with little to no rigorous theory to guide experimentation. Despite this fact, practitioners have been able to guide their future experimentation via observations from previous large-scale empirical investigations. However, alluding to Plato's Allegory of the cave, it is likely that the observations which form the field's notion of reality are but shadows representing fragments of that reality. In this work, we propose a theoretical framework which attempts to answer what exists outside of the cave. To the theorist, we provide a framework which is mathematically rigorous and leaves open many interesting ideas for future exploration. To the practitioner, we provide a framework whose results are very intuitive, general, and which will help form principles to guide future investigations. Concretely, we provide a theoretical framework rooted in Bayesian statistics and Shannon's information theory which is general enough to unify the analysis of many phenomena in machine learning. Our framework characterizes the performance of an optimal Bayesian learner, which considers the fundamental limits of information. Throughout this work, we derive very general theoretical results and apply them to derive insights specific to settings ranging from data which is independently and identically distributed under an unknown distribution, to data which is sequential, to data which exhibits hierarchical structure amenable to meta-learning. We conclude with a section dedicated to characterizing the performance of misspecified algorithms. These results are exciting and particularly relevant as we strive to overcome increasingly difficult machine learning challenges in this endlessly complex world.

Structure Learning via Mutual Information

Mutual Information Multinomial Estimation

Approximating mutual information of high-dimensional variables using learned representations

Exploiting Mutual Information for Substructure-Aware Graph Representation Learning

Mutual-information-inspired heuristics for constraint-based causal structure learning

Learning non-parametric Markov networks with mutual information

Information-Theoretic Foundations for Machine Learning

A robust estimator of mutual information for deep learning interpretability

Constructing Enhanced Mutual Information for Online Class-Incremental Learning

Learning Bayesian Network Structures Based on Mutual Information

Mutual Information calculation on different appearances

Mutual Information Analysis in Multimodal Learning Systems

DeepMI: A Mutual Information Based Framework For Unsupervised Deep Learning of Tasks

An information-matching approach to optimal experimental design and active learning

Mutual information analysis on non-stationary neuron importance for brain machine interfaces.

Towards a rigorous analysis of mutual information in contrastive learning

Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learning

Explaining Representation by Mutual Information

Feature selection based on Mutual Information in supervised learning

Understanding Encoder-Decoder Structures in Machine Learning Using Information Measures

An Adversarial Training-Based Mutual Information Constraint Method