Abstract:In this paper, we present a novel parallel implementation for training Gradient Boosting Decision Trees (GBDTs) on Graphics Processing Units (GPUs). Thanks to the excellent results on classification/regression and the open sourced libraries such as XGBoost, GBDTs have become very popular in recent years and won many awards in machine learning and data mining competitions. Although GPUs have demonstrated their success in accelerating many machine learning applications, it is challenging to develop an efficient GPU-based GBDT algorithm. The key challenges include irregular memory accesses, many sorting operations with small inputs and varying data parallel granularities in tree construction. To tackle these challenges on GPUs, we propose various novel techniques including (i) Run-length Encoding compression and thread/block workload dynamic allocation, (ii) data partitioning based on stable sort, and fast and memory efficient attribute ID lookup in node splitting, (iii) finding approximate split points using two-stage histogram building, (iv) building histograms with the aware of sparsity and exploiting histogram subtraction to reduce histogram building workload, (v) reusing intermediate training results for efficient gradient computation, and (vi) exploiting multiple GPUs to handle larger data sets efficiently. Our experimental results show that our algorithm named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ThunderGBM</italic> can be 10x times faster than the state-of-the-art libraries (i.e., XGBoost, LightGBM and CatBoost) running on a relatively high-end workstation of 20 CPU cores. In comparison with the libraries on GPUs, ThunderGBM can handle higher dimensional problems which the libraries become extremely slow or simply fail. For the data sets the existing libraries on GPUs can handle, ThunderGBM achieves up to 10 times speedup on the same hardware, which demonstrates the significance of our GPU optimizations. Moreover, the models trained by ThunderGBM are identical to those trained by XGBoost, and have similar quality as those trained by LightGBM and CatBoost.

TRBoost: a generic gradient boosting machine based on trust-region method

Accelerating Gradient Boosting Machine

HardGBM: A Framework for Accurate and Hardware-Efficient Gradient Boosting Machines.

Soft Gradient Boosting Machine

TencentBoost: A Gradient Boosting Tree System with Parameter Server

Beyond Nonconvexity: A Universal Trust-Region Method with New Analyses

Gradient Boosting With Piece-Wise Linear Regression Trees

A Weighted-Resampling Based Transfer Learning Algorithm.

SecureBoost+: Large Scale and High-Performance Vertical Federated Gradient Boosting Decision Tree

Boosting Gradient Ascent for Continuous DR-submodular Maximization

TransBoost: A Boosting-Tree Kernel Transfer Learning Algorithm for Improving Financial Inclusion

Poster: gbdt-rs: Fast and Trustworthy Gradient Boosting Decision Tree

Exploiting GPUs for Efficient Gradient Boosting Decision Tree Training

RoNGBa: A Robustly Optimized Natural Gradient Boosting Training Approach with Leaf Number Clipping

Historical Gradient Boosting Machine.

Gradient Norm Regularization Second-Order Algorithms for Solving Nonconvex-Strongly Concave Minimax Problems

A New Formulation of Gradient Boosting

Probabilistic Gradient Boosting Machines for Large-Scale Probabilistic Regression

Improved TrAdaBoost and Its Application to Transaction Fraud Detection.

XGrad: Boosting Gradient-Based Optimizers With Weight Prediction

UTBoost: Gradient Boosted Decision Trees for Uplift Modeling