Abstract:LogitBoost is a popular Boosting variant that can be applied to either binary or multi-class classification. From a statistical viewpoint LogitBoost can be seen as additive tree regression by minimizing the Logistic loss. Following this setting, it is still non-trivial to devise a sound multi-class LogitBoost compared with to devise its binary counterpart. The difficulties are due to two important factors arising in multiclass Logistic loss. The first is the invariant property implied by the Logistic loss, causing the optimal classifier output being not unique, i.e. adding a constant to each component of the output vector won’t change the loss value. The second is the density of the Hessian matrices that arise when computing tree node split gain and node value fittings. Oversimplification of this learning problem can lead to degraded performance. For example, the original LogitBoost algorithm is outperformed by ABC-LogitBoost thanks to the latter’s more careful treatment of the above two factors. In this paper we propose new techniques to address the two main difficulties in multiclass LogitBoost setting: (1) we adopt a vector tree model (i.e. each node value is vector) where the unique classifier output is guaranteed by adding a sum-to-zero constraint, and (2) we use an adaptive block coordinate descent that exploits the dense Hessian when computing tree split gain and node values. Higher classification accuracy and faster convergence rates are observed for a range of public data sets when compared to both the original and the ABC-LogitBoost implementations. We also discuss another possibility to cope with LogitBoost’s dense Hessian matrix. We derive a loss similar to the multi-class Logistic loss but which guarantees a diagonal Hessian matrix. While this makes the optimization (by Newton descent) easier we unfortunately observe degraded performance for this modification. We argue that working with the dense Hessian is likely unavoidable, therefore making techniques like those proposed in this paper necessary for efficient implementations.

Towards Optimal Discriminating Order for Multiclass Classification

A Multi-Class Large Margin Classifier

Novel Design of Decision-Tree-Based Support Vector Machines Multi-Class Classifier

Strong Optimal Classification Trees

A scalable stage-wise approach to large-margin multi-class loss based boosting

Mixed-Integer Linear Optimization for Semi-Supervised Optimal Classification Trees

A Scalable Stagewise Approach to Large-Margin Multiclass Loss-Based Boosting

Totally Corrective Multiclass Boosting with Binary Weak Learners

BooleanOCT: Optimal Classification Trees based on multivariate Boolean Rules

The return of AdaBoost.MH: multi-class Hamming trees

Large-margin Feature Selection for Monotonic Classification

Mixed integer linear optimization formulations for learning optimal binary classification trees

Optimal Mixed Integer Linear Optimization Trained Multivariate Classification Trees

Boosting Through Optimization of Margin Distributions

Boosting Trees for Cost-Sensitive Classifications.

for Further Innproving the Performance of Decision Tree Learning

Boosting Bootstrap Fld Subspaces In Classification

A scalable direct formulation for multi-class boosting

Efficient Tree Classifiers for Large Scale Datasets.

A Direct Approach to Multi-class Boosting and Extensions

An improved multiclass LogitBoost using adaptive-one-vs-one