Abstract:AdaBoost is one of the most popular ML algorithms. It is simple to implement and often found very effective by practitioners, while still being mathematically elegant and theoretically sound. AdaBoost's interesting behavior in practice still puzzles the ML community. We address the algorithm's stability and establish multiple convergence properties of "Optimal AdaBoost," a term coined by Rudin, Daubechies, and Schapire in 2004. We prove, in a reasonably strong computational sense, the almost universal existence of time averages, and with that, the convergence of the classifier itself, its generalization error, and its resulting margins, among many other objects, for fixed data sets under arguably reasonable conditions. Specifically, we frame Optimal AdaBoost as a dynamical system and, employing tools from ergodic theory, prove that, under a condition that Optimal AdaBoost does not have ties for best weak classifier eventually, a condition for which we provide empirical evidence from high dimensional real-world datasets, the algorithm's update behaves like a continuous map. We provide constructive proofs of several arbitrarily accurate approximations of Optimal AdaBoost; prove that they exhibit certain cycling behavior in finite time, and that the resulting dynamical system is ergodic; and establish sufficient conditions for the same to hold for the actual Optimal-AdaBoost update. We believe that our results provide reasonably strong evidence for the affirmative answer to two open conjectures, at least from a broad computational-theory perspective: AdaBoost always cycles and is an ergodic dynamical system. We present empirical evidence that cycles are hard to detect while time averages stabilize quickly. Our results ground future convergence-rate analysis and may help optimize generalization ability and alleviate a practitioner's burden of deciding how long to run the algorithm.

AdaBoost typical Algorithm and its application research

AdaBoost-based Multi-attribute Classification Technology and Its Application

Multi-class AdaBoost ELM

Research of the Real Adaboost Algorithm

A Revised AdaBoost Algorithm——AD AdaBoost

Two-Stage Multi-Class Adaboost for Facial Expression Recognition

Z-Adaboost: Boosting 2-thresholded weak classifiers for object detection

Overview of AdaBoost : Reconciling its views to better understand its dynamics

The success of AdaBoost and its application in portfolio management

Feature Learning Viewpoint of AdaBoost and a New Algorithm

Experiments with a New Boosting Algorithm

The Application Ofadaboost in Customer Churn Prediction

A new boosting algorithm based on dual averaging scheme

A Regression Algorithm Based on AdaBoost

Untangling AdaBoost-based Cost-Sensitive Classification. Part I: Theoretical Perspective

The Evolution of Boosting Algorithms - From Machine Learning to Statistical Modelling

A Theoretical Approach To Construct Highly Discriminative Features With Application In Adaboost

Theory deduction of AdaBoost classification

Optimization of the aggregation in AdaBoost algorithm by particle swarm optimization and its application in classification problems

On the Convergence Properties of Optimal AdaBoost

Real-time face detection with self-adaptive cost-sensitive AdaBoost