Abstract:LogitBoost is a popular Boosting variant that can be applied to either binary or multi-class classification. From a statistical viewpoint LogitBoost can be seen as additive tree regression by minimizing the Logistic loss. Following this setting, it is still non-trivial to devise a sound multi-class LogitBoost compared with to devise its binary counterpart. The difficulties are due to two important factors arising in multiclass Logistic loss. The first is the invariant property implied by the Logistic loss, causing the optimal classifier output being not unique, i.e. adding a constant to each component of the output vector won’t change the loss value. The second is the density of the Hessian matrices that arise when computing tree node split gain and node value fittings. Oversimplification of this learning problem can lead to degraded performance. For example, the original LogitBoost algorithm is outperformed by ABC-LogitBoost thanks to the latter’s more careful treatment of the above two factors. In this paper we propose new techniques to address the two main difficulties in multiclass LogitBoost setting: (1) we adopt a vector tree model (i.e. each node value is vector) where the unique classifier output is guaranteed by adding a sum-to-zero constraint, and (2) we use an adaptive block coordinate descent that exploits the dense Hessian when computing tree split gain and node values. Higher classification accuracy and faster convergence rates are observed for a range of public data sets when compared to both the original and the ABC-LogitBoost implementations. We also discuss another possibility to cope with LogitBoost’s dense Hessian matrix. We derive a loss similar to the multi-class Logistic loss but which guarantees a diagonal Hessian matrix. While this makes the optimization (by Newton descent) easier we unfortunately observe degraded performance for this modification. We argue that working with the dense Hessian is likely unavoidable, therefore making techniques like those proposed in this paper necessary for efficient implementations.

A Convergence Rate Analysis for LogitBoost, MART and Their Variant.

An improved multiclass LogitBoost using adaptive-one-vs-one

The Rate of Convergence of AdaBoost

AOSO-LogitBoost: Adaptive One-Vs-One LogitBoost for Multi-Class Problem

Logistic Regression, AdaBoost and Bregman Distances

On the Convergence Properties of Optimal AdaBoost

High-Dimensional $L_2$Boosting: Rate of Convergence

Analysis of boosting algorithms using the smooth margin function

A Fast Sampling Gradient Tree Boosting Framework

BOOSTED NONPARAMETRIC HAZARDS WITH TIME-DEPENDENT COVARIATES

Optimization by gradient boosting

An empirical study of the convergence of regionboost

Faster Margin Maximization Rates for Generic and Adversarially Robust Optimization Methods

Infinitesimal gradient boosting

A Novel Family of Boosted Online Regression Algorithms with Strong Theoretical Bounds

pGMM Kernel Regression and Comparisons with Boosted Trees

An Empirical Evaluation of Four Algorithms for Multi-Class Classification: Mart, ABC-Mart, Robust LogitBoost, and ABC-LogitBoost

Optimal AdaBoost Converges

Tree-Structured Boosting: Connections Between Gradient Boosted Stumps and Full Decision Trees

Robust boosting for regression problems

Surrogate Maximization/minimization Algorithms for AdaBoost and the Logistic Regression Model.