Abstract:The quintessential learning algorithm of empirical risk minimization (ERM) is known to fail in various settings for which uniform convergence does not characterize learning. It is therefore unsurprising that the practice of machine learning is rife with considerably richer algorithmic techniques for successfully controlling model capacity. Nevertheless, no such technique or principle has broken away from the pack to characterize optimal learning in these more general settings. The purpose of this work is to characterize the role of regularization in perhaps the simplest setting for which ERM fails: multiclass learning with arbitrary label sets. Using one-inclusion graphs (OIGs), we exhibit optimal learning algorithms that dovetail with tried-and-true algorithmic principles: Occam's Razor as embodied by structural risk minimization (SRM), the principle of maximum entropy, and Bayesian reasoning. Most notably, we introduce an optimal learner which relaxes structural risk minimization on two dimensions: it allows the regularization function to be "local" to datapoints, and uses an unsupervised learning stage to learn this regularizer at the outset. We justify these relaxations by showing that they are necessary: removing either dimension fails to yield a near-optimal learner. We also extract from OIGs a combinatorial sequence we term the Hall complexity, which is the first to characterize a problem's transductive error rate exactly. Lastly, we introduce a generalization of OIGs and the transductive learning setting to the agnostic case, where we show that optimal orientations of Hamming graphs -- judged using nodes' outdegrees minus a system of node-dependent credits -- characterize optimal learners exactly. We demonstrate that an agnostic version of the Hall complexity again characterizes error rates exactly, and exhibit an optimal learner using maximum entropy programs.

Statistical Learning Theory and Occam's Razor: The Core Argument

Simple Models in Complex Worlds: Occam’s Razor and Statistical Learning Theory

In-context learning and Occam's razor

Razor sharp: The role of Occam's razor in science

OccamNets: Mitigating Dataset Bias by Favoring Simpler Hypotheses

Inside Ockham's razor: A mechanism driving preferences for simpler explanations

Achieving Occam's Razor: Deep Learning for Optimal Model Reduction

A Simple Generative Model of Logical Reasoning and Statistical Learning

The Fragility Of Standard Inferential Approaches In Principal Stratification Models Relative To Direct Likelihood Approaches

Statistical relational learning and neuro-symbolic AI: what does first-order logic offer?

Statistical Learning Theory: Models, Concepts, and Results

Its All on the Square- The Importance of the Sum of Squares and Making the General Linear Model Simple

Occam's hammer: a link between randomized learning and multiple testing FDR control

The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning

Identifying Spurious Biases Early in Training through the Lens of Simplicity Bias

Regularization and Optimal Multiclass Learning

Classical Statistical (In-Sample) Intuitions Don't Generalize Well: A Note on Bias-Variance Tradeoffs, Overfitting and Moving from Fixed to Random Designs

A Path to Simpler Models Starts With Noise

Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihood

Sharpening Occam's Razor with Quantum Mechanics

Fairness, Accuracy, and Unreliable Data