Abstract:The quintessential learning algorithm of empirical risk minimization (ERM) is known to fail in various settings for which uniform convergence does not characterize learning. It is therefore unsurprising that the practice of machine learning is rife with considerably richer algorithmic techniques for successfully controlling model capacity. Nevertheless, no such technique or principle has broken away from the pack to characterize optimal learning in these more general settings. The purpose of this work is to characterize the role of regularization in perhaps the simplest setting for which ERM fails: multiclass learning with arbitrary label sets. Using one-inclusion graphs (OIGs), we exhibit optimal learning algorithms that dovetail with tried-and-true algorithmic principles: Occam's Razor as embodied by structural risk minimization (SRM), the principle of maximum entropy, and Bayesian reasoning. Most notably, we introduce an optimal learner which relaxes structural risk minimization on two dimensions: it allows the regularization function to be "local" to datapoints, and uses an unsupervised learning stage to learn this regularizer at the outset. We justify these relaxations by showing that they are necessary: removing either dimension fails to yield a near-optimal learner. We also extract from OIGs a combinatorial sequence we term the Hall complexity, which is the first to characterize a problem's transductive error rate exactly. Lastly, we introduce a generalization of OIGs and the transductive learning setting to the agnostic case, where we show that optimal orientations of Hamming graphs -- judged using nodes' outdegrees minus a system of node-dependent credits -- characterize optimal learners exactly. We demonstrate that an agnostic version of the Hall complexity again characterizes error rates exactly, and exhibit an optimal learner using maximum entropy programs.

Classification with Non-I.i.d. Sampling

Concentration Estimates for Learning with Unbounded Sampling

Concentration Estimates for Learning with ℓ1-Regularizer and Data Dependent Hypothesis Spaces

Learning Theory Estimates for Coefficient-Based Regularized Regression

Learning with coefficient-based regularization and ℓ 1 −penalty

Online Regression with Unbounded Sampling.

Convergence Conditions of Online Regularized Statistical Learning in Reproducing Kernel Hilbert Space With Non-Stationary Data

Convergence of Unregularized Online Learning Algorithms

Guaranteed classification via regularized similarity learning.

Relevant Sampling in a reproducing kernel subspace of Orlicz space

Online Regularized Learning with Pairwise Loss Functions

Generalization Error Bounds for Learning under Censored Feedback

Nonuniform sampling, reproducing kernels, and the associated Hilbert spaces

The Generalization Ability of SVM Classification Based on Markov Sampling

Classification with Valid and Adaptive Coverage

Nonlinear Online Classification Algorithm with Probability Margin

Regularization and Optimal Multiclass Learning

Generalization errors of Laplacian regularized least squares regression

On Coresets for Regularized Loss Minimization

Inference with non-differentiable surrogate loss in a general high-dimensional classification framework

Optimal rate of the regularized regression learning algorithm