Abstract:In Hyperparameter Optimization (HPO), only the hyperparameter configuration with the best performance is chosen after performing several trials, then, discarding the effort of training all the models with every hyperparameter configuration trial and performing an ensemble of all them. This ensemble consists of simply averaging the model predictions or weighting the models by a certain probability. Recently, other more sophisticated ensemble strategies, such as the Caruana method or the stacking strategy has been proposed. On the one hand, the Caruana method performs well in HPO ensemble, since it is not affected by the effects of multicollinearity, which is prevalent in HPO. It just computes the average over a subset of predictions with replacement. But it does not benefit from the generalization power of a learning process. On the other hand, stacking methods include a learning procedure since a meta-learner is required to perform the ensemble. Yet, one hardly finds advice about which meta-learner is adequate. Besides, some meta-learners may suffer from the effects of multicollinearity or need to be tuned to reduce them. This paper explores meta-learners for stacking ensemble in HPO, free of hyperparameter tuning, able to reduce the effects of multicollinearity and considering the ensemble learning process generalization power. At this respect, the boosting strategy seems promising as a stacking meta-learner. In fact, it completely removes the effects of multicollinearity. This paper also proposes an implicit regularization in the classical boosting method and a novel non-parametric stop criterion suitable only for boosting and specifically designed for HPO. The synergy between these two improvements over boosting exhibits competitive and promising predictive power performance compared to other existing meta-learners and ensemble approaches for HPO other than the stacking ensemble.

Homotopy Regularization for Boosting

Totally Corrective Boosting for Regularized Risk Minimization

Hyperspherical Consistency Regularization.

Fully corrective boosting with arbitrary loss and regularization.

BOOSTED NONPARAMETRIC HAZARDS WITH TIME-DEPENDENT COVARIATES

An Aggressive Reduction on the Complexity of Optimization for Non-Strongly Convex Objectives

On the Dual Formulation of Boosting Algorithms

Homotopy continuation approaches for robust SV classification and regression

On Stochastic Primal-Dual Hybrid Gradient Approach for Compositely Regularized Minimization.

Efficient Hyper-parameter Optimization with Cubic Regularization.

Improved Optimization Methods For Regularized Optimal Transport

An algorithmic view of $\ell_2$ regularization and some path-following algorithms

Totally Corrective Multiclass Boosting with Binary Weak Learners

A Direct Formulation for Totally-Corrective Multi-Class Boosting

Regularized boosting with an increasing coefficient magnitude stop criterion as meta-learner in hyperparameter optimization stacking ensemble

The Role of Fine-tuning: Transfer Learning for High-dimensional M-estimators with Decomposable Regularizers

High-Dimensional $L_2$Boosting: Rate of Convergence

Local Boosting for Weakly-Supervised Learning

Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbation

Tikhonov Regularization for Stochastic Non-Smooth Convex Optimization in Hilbert Spaces

A new boosting algorithm based on dual averaging scheme