Abstract:Gradient Boosting Machines (GBMs) have achieved remarkable success in effectively solving a wide range of problems by leveraging Taylor expansions in functional space. Second-order Taylor-based GBMs, such as XGBoost rooted in Newton's method, consistently yield state-of-the-art results in practical applications. However, it is important to note that the loss functions used in second-order GBMs must strictly adhere to convexity requirements, specifically requiring a positive definite Hessian of the loss. This restriction significantly narrows the range of objectives, thus limiting the application scenarios. In contrast, first-order GBMs are based on the first-order gradient optimization method, enabling them to handle a diverse range of loss functions. Nevertheless, their performance may not always meet expectations. To overcome this limitation, we introduce Trust-region Boosting (TRBoost), a new and versatile Gradient Boosting Machine that combines the strengths of second-order GBMs and the versatility of first-order GBMs. In each iteration, TRBoost employs a constrained quadratic model to approximate the objective and applies the Trust-region algorithm to obtain a new learner. Unlike GBMs based on Newton's method, TRBoost does not require a positive definite Hessian, enabling its application to more loss functions while achieving competitive performance similar to second-order algorithms. Convergence analysis and numerical experiments conducted in this study confirm that TRBoost exhibits similar versatility to first-order GBMs and delivers competitive results compared to second-order GBMs. Overall, TRBoost presents a promising approach that achieves a balance between performance and generality, rendering it a valuable addition to the toolkit of machine learning practitioners.

Generalized XGBoost Method

LinXGBoost: Extension of XGBoost to Generalized Local Linear Models

A XGBoost risk model via feature selection and Bayesian hyper-parameter optimization

Multi-Target XGBoostLSS Regression

Extreme Gradient Boosting with Squared Logistic Loss Function

Adaptive generalized conditional gradient method for multiobjective optimization

BOOSTED NONPARAMETRIC HAZARDS WITH TIME-DEPENDENT COVARIATES

XGBoost: A Scalable Tree Boosting System

Boosting in the presence of outliers: adaptive classification with non-convex loss functions

Composite Quantile Regression With XGBoost Using the Novel Arctan Pinball Loss

Optimization by gradient boosting

Greedy function approximation: A gradient boosting machine.

Functional Frank-Wolfe Boosting for General Loss Functions

TRBoost: a generic gradient boosting machine based on trust-region method

Research and application of XGBoost in imbalanced data

Random gradient boosting for predicting conditional quantiles

A comparative analysis of gradient boosting algorithms

Generalised Boosted Forests

GEV Regression with Convex Loss Applied to Imbalanced Binary Classification.

Predicting Chinese stock market using XGBoost multi-objective optimization with optimal weighting

Imbalance-XGBoost: leveraging weighted and focal losses for binary label-imbalanced classification with XGBoost