Abstract:In recent years, an interest appeared in integrating various optimization algorithms in machine learning. We study the potential of ensemble learning in classification tasks and how to efficiently decompose the underlying optimization problem. Ensemble learning has become popular for machine learning applications and it is particularly interesting from an optimization perspective due to its resemblance to column generation. The challenge for learning is not only to obtain a good fit for the training data set, but also good generalization, such that the classifier is generally applicable. Deep networks have the drawback that they require a lot of computational effort to get to an accurate classification. Ensemble learning can combine various weak learners, which individually require less computational time. We consider binary classification problems studying a three-phase algorithm. After initializing a set of base learners refined by a bootstrapping approach, base learners are generated using the solution of an linear programming (LP) master problem and then solving a machine learning sub-problem regarding a reduced data set, which can be viewed as a so-called pricing problem. We theoretically show that the algorithm computes an optimal ensemble model in the convex hull of a given model space. The implementation of the algorithm is part of an ensemble learning framework called decolearn . Numerical experiments with CIFAR-10 data set show that the base learners are diverse and that both the training and generalization error are reduced after several iterations.

Fast training of effective multi-class boosting using coordinate descent optimization

UniGrad-FS: Unified Gradient Projection with Flatter Sharpness for Continual Learning

A Direct Approach to Multi-class Boosting and Extensions

A scalable direct formulation for multi-class boosting

A scalable stage-wise approach to large-margin multi-class loss based boosting

A Scalable Stagewise Approach to Large-Margin Multiclass Loss-Based Boosting

Online coordinate boosting

Totally-Corrective Multi-Class Boosting

Multi-class Image Classification Based on Fast Stochastic Gradient Boosting.

Even Faster Accelerated Coordinate Descent Using Non-Uniform Sampling

On Optimizing Ensemble Models using Column Generation

Gradient Descent Optimization in Deep Learning Model Training Based on Multistage and Method Combination Strategy

A new boosting algorithm based on dual averaging scheme

A Hybrid Method of Combinatorial Search and Coordinate Descent for Discrete Optimization

Condensed Gradient Boosting

Parallel Coordinate Descent Newton for Large-scale L1-Regularized Minimization.

Accelerated Gradient-free Neural Network Training by Multi-convex Alternating Optimization

A Multi-task Learning Approach by Combining Derivative-Free and Gradient Methods.

A Fast Sampling Gradient Tree Boosting Framework

Multi-Task Learning with Multi-Task Optimization

A Direct Formulation for Totally-Corrective Multi-Class Boosting