Abstract:AdaBoost is one of the most popular ML algorithms. It is simple to implement and often found very effective by practitioners, while still being mathematically elegant and theoretically sound. AdaBoost's interesting behavior in practice still puzzles the ML community. We address the algorithm's stability and establish multiple convergence properties of "Optimal AdaBoost," a term coined by Rudin, Daubechies, and Schapire in 2004. We prove, in a reasonably strong computational sense, the almost universal existence of time averages, and with that, the convergence of the classifier itself, its generalization error, and its resulting margins, among many other objects, for fixed data sets under arguably reasonable conditions. Specifically, we frame Optimal AdaBoost as a dynamical system and, employing tools from ergodic theory, prove that, under a condition that Optimal AdaBoost does not have ties for best weak classifier eventually, a condition for which we provide empirical evidence from high dimensional real-world datasets, the algorithm's update behaves like a continuous map. We provide constructive proofs of several arbitrarily accurate approximations of Optimal AdaBoost; prove that they exhibit certain cycling behavior in finite time, and that the resulting dynamical system is ergodic; and establish sufficient conditions for the same to hold for the actual Optimal-AdaBoost update. We believe that our results provide reasonably strong evidence for the affirmative answer to two open conjectures, at least from a broad computational-theory perspective: AdaBoost always cycles and is an ergodic dynamical system. We present empirical evidence that cycles are hard to detect while time averages stabilize quickly. Our results ground future convergence-rate analysis and may help optimize generalization ability and alleviate a practitioner's burden of deciding how long to run the algorithm.

A note on the convergence of a class of adaptive optimal identifiers

A Robust Output Error Identifier for Continuous-Time Systems

On Stability and Convergence of Suboptimal Estimation for Systems over Lossy Networks Without Acknowledgement

On the Convergence Properties of Optimal AdaBoost

Axioms of adaptivity

A Robust Strictly Passive Output—Error Identifier for Continuous—Time Systems

Identifiability Implies Robust, Globally Exponentially Convergent On-line Parameter Estimation: Application to Model Reference Adaptive Control

Adaptive Neural Network Nonparametric Identifier With Normalized Learning Laws

A New Adaptive Non-singleton Fuzzy Identifier

On the convergence of adaptive approximations for stochastic differential equations

A new approach to convergence analysis of iterative models with optimal error bounds

Adaptive Alternating Minimization Algorithms

On an integral variant of incremental input/output-to-state stability and its use as a notion of nonlinear detectability

An adaptation theory for nonparametric confidence intervals

Demistifying Inference after Adaptive Experiments

A UNIFIED ANALYSIS OF QUASI-OPTIMAL CONVERGENCE FOR ADAPTIVE MIXED FINITE ELEMENT METHODS

Adaptive confidence bands in the nonparametric fixed design regression model

Convergence and concentration properties of constant step-size SGD through Markov chains

Uniform exponential convergence of SAA with AMIS and asymptotics of its optimal value

A general framework of Riemannian adaptive optimization methods with a convergence analysis

Hybrid Persistency of Excitation in Adaptive Estimation for Hybrid Systems