Abstract:This paper establishes a precise high-dimensional asymptotic theory for boosting on separable data, taking statistical and computational perspectives. We consider a high-dimensional setting where the number of features (weak learners) $p$ scales with the sample size $n$, in an overparametrized regime. Under a class of statistical models, we provide an exact analysis of the generalization error of boosting when the algorithm interpolates the training data and maximizes the empirical $\ell_1$-margin. Further, we explicitly pin down the relation between the boosting test error and the optimal Bayes error, as well as the proportion of active features at interpolation (with zero initialization). In turn, these precise characterizations answer certain questions raised in \cite{breiman1999prediction, schapire1998boosting} surrounding boosting, under assumed data generating processes. At the heart of our theory lies an in-depth study of the maximum-$\ell_1$-margin, which can be accurately described by a new system of non-linear equations; to analyze this margin, we rely on Gaussian comparison techniques and develop a novel uniform deviation argument. Our statistical and computational arguments can handle (1) any finite-rank spiked covariance model for the feature distribution and (2) variants of boosting corresponding to general $\ell_q$-geometry, $q \in [1, 2]$. As a final component, via the Lindeberg principle, we establish a universality result showcasing that the scaled $\ell_1$-margin (asymptotically) remains the same, whether the covariates used for boosting arise from a non-linear random feature model or an appropriately linearized model with matching moments.

Infinitesimal gradient boosting

Optimization by gradient boosting

Behavior of linear L2-boosting algorithms in the vanishing learning rate asymptotic

The vanishing learning rate asymptotic for linear L2-boosting

Boulevard: Regularized Stochastic Gradient Boosted Trees and Their Limiting Distribution

A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-$\ell_1$-Norm Interpolated Classifiers

Limit Theorems for Stochastic Gradient Descent with Infinite Variance

A Fast Sampling Gradient Tree Boosting Framework

Gradient boosting machine with partially randomized decision trees

Gradient and Newton boosting for classification and regression

A Convergence Rate Analysis for LogitBoost, MART and Their Variant.

Individually Fair Gradient Boosting

Gradient boosting for extreme quantile regression

Wasserstein Gradient Boosting: A Framework for Distribution-Valued Supervised Learning

The column measure and Gradient-Free Gradient Boosting

Robust boosting for regression problems

Proximal boosting: aggregating weak learners to minimize non-differentiable losses

Greedy function approximation: A gradient boosting machine.

Incremental Majorization-Minimization Optimization with Application to Large-Scale Machine Learning

Logistic Regression, AdaBoost and Bregman Distances