990 LightGBM: A highly efficient gradient boosting decision tree

G Ke, Q Meng, T Finley, T Wang, W Chen, W Ma, Q Ye, TY Liu

2017-01-01

Abstract:

What problem does this paper attempt to address?

DimBoost

Jiawei Jiang,Bin Cui,Ce Zhang,Fangcheng Fu

DOI: https://doi.org/10.1145/3183713.3196892

2018-01-01

Abstract:Gradient boosting decision tree (GBDT) is one of the most popular machine learning models widely used in both academia and industry. Although GBDT has been widely supported by existing systems such as XGBoost, LightGBM, and MLlib, one system bottleneck appears when the dimensionality of the data becomes high. As a result, when we tried to support our industrial partner on datasets of the dimension up to 330K, we observed suboptimal performance for all these aforementioned systems. In this paper, we ask "Can we build a scalable GBDT training system whose performance scales better with respect to dimensionality of the data?"
Gradient Boosting With Piece-Wise Linear Regression Trees

Yu Shi,Jian Li,Zhize Li

DOI: https://doi.org/10.48550/arXiv.1802.05640

2019-06-26

Abstract:Gradient Boosted Decision Trees (GBDT) is a very successful ensemble learning algorithm widely used across a variety of applications. Recently, several variants of GBDT training algorithms and implementations have been designed and heavily optimized in some very popular open sourced toolkits including XGBoost, LightGBM and CatBoost. In this paper, we show that both the accuracy and efficiency of GBDT can be further enhanced by using more complex base learners. Specifically, we extend gradient boosting to use piecewise linear regression trees (PL Trees), instead of piecewise constant regression trees, as base learners. We show that PL Trees can accelerate convergence of GBDT and improve the accuracy. We also propose some optimization tricks to substantially reduce the training time of PL Trees, with little sacrifice of accuracy. Moreover, we propose several implementation techniques to speedup our algorithm on modern computer architectures with powerful Single Instruction Multiple Data (SIMD) parallelism. The experimental results show that GBDT with PL Trees can provide very competitive testing accuracy with comparable or less training time.

Machine Learning
Quantized Training of Gradient Boosting Decision Trees

Yu Shi,Guolin Ke,Zhuoming Chen,Shuxin Zheng,Tie-Yan Liu

DOI: https://doi.org/10.48550/arXiv.2207.09682

IF: 5.414

2022-07-20

Machine Learning

Abstract:Recent years have witnessed significant success in Gradient Boosting Decision Trees (GBDT) for a wide range of machine learning applications. Generally, a consensus about GBDT's training algorithms is gradients and statistics are computed based on high-precision floating points. In this paper, we investigate an essentially important question which has been largely ignored by the previous literature: how many bits are needed for representing gradients in training GBDT? To solve this mystery, we propose to quantize all the high-precision gradients in a very simple yet effective way in the GBDT's training algorithm. Surprisingly, both our theoretical analysis and empirical studies show that the necessary precisions of gradients without hurting any performance can be quite low, e.g., 2 or 3 bits. With low-precision gradients, most arithmetic operations in GBDT training can be replaced by integer operations of 8, 16, or 32 bits. Promisingly, these findings may pave the way for much more efficient training of GBDT from several aspects: (1) speeding up the computation of gradient statistics in histograms; (2) compressing the communication cost of high-precision statistical information during distributed training; (3) the inspiration of utilization and development of hardware architectures which well support low-precision computation for GBDT training. Benchmarked on CPUs, GPUs, and distributed clusters, we observe up to 2$\times$ speedup of our simple quantization strategy compared with SOTA GBDT systems on extensive datasets, demonstrating the effectiveness and potential of the low-precision training of GBDT. The code will be released to the official repository of LightGBM.
MT-GBM: A Multi-Task Gradient Boosting Machine with Shared Decision Trees

ZhenZhe Ying,Zhuoer Xu,Zhifeng Li,Weiqiang Wang,Changhua Meng

DOI: https://doi.org/10.48550/arXiv.2201.06239

2022-01-24

Abstract:Despite the success of deep learning in computer vision and natural language processing, Gradient Boosted Decision Tree (GBDT) is yet one of the most powerful tools for applications with tabular data such as e-commerce and FinTech. However, applying GBDT to multi-task learning is still a challenge. Unlike deep models that can jointly learn a shared latent representation across multiple tasks, GBDT can hardly learn a shared tree structure. In this paper, we propose Multi-task Gradient Boosting Machine (MT-GBM), a GBDT-based method for multi-task learning. The MT-GBM can find the shared tree structures and split branches according to multi-task losses. First, it assigns multiple outputs to each leaf node. Next, it computes the gradient corresponding to each output (task). Then, we also propose an algorithm to combine the gradients of all tasks and update the tree. Finally, we apply MT-GBM to LightGBM. Experiments show that our MT-GBM improves the performance of the main task significantly, which means the proposed MT-GBM is efficient and effective.

Machine Learning
HarpGBDT: Optimizing Gradient Boosting Decision Tree for Parallel Efficiency

Bo Peng,Langshi Chen,Jiayu Li,Miao Jiang,Selahattin Akkas,Egor Smirnov,Ruslan Israfilov,Sergey Khekhnev,Andrey Nikolaev,Judy Qiu

DOI: https://doi.org/10.1109/cluster.2019.8890990

2019-01-01

Abstract:Gradient Boosting Decision Tree (GBDT) is a widely used machine learning algorithm, whose training involves both irregular computation and random memory access and is challenging for system optimizations. In this paper, we conduct a comprehensive performance analysis of two state-of-the-art systems, XGBoost and LightGBM. They represent two typical parallel implementations for GBDT; one is data parallel and the other one is parallel over features. Substantial thread synchronization overhead, as well as the inefficiency of random memory access, is identified. We propose HarpGBDT, a new GBDT system designed from the perspective of parallel efficiency optimization. Firstly, we adopt a new tree growth method that selects the top K candidates of tree nodes to enable the use of more levels of parallelism without sacrificing the algorithm's accuracy. Secondly, we organize the training data and model data in blocks and propose a block-wise approach as a general model that enables the exploration of various parallelism options. Thirdly, we propose a mixed mode to utilize the advantages of a different mode of parallelism in different phases of training. By changing the configuration of the block size and parallel mode, HarpGBDT is able to attain better parallel efficiency. By extensive experiments on four datasets with different statistical characteristics on the Intel(R) Xeon(R) E5-2699 server, HarpGBDT on average performs 8x faster than XGBoost and 2.6x faster than LightGBM.
LightGBM: accelerated genomically designed crop breeding through ensemble learning

Jun Yan,Yuetong Xu,Qian Cheng,Shuqin Jiang,Qian Wang,Yingjie Xiao,Chuang Ma,Jianbing Yan,Xiangfeng Wang

DOI: https://doi.org/10.1186/s13059-021-02492-y

IF: 17.906

2021-09-20

Genome Biology

Abstract:Abstract LightGBM is an ensemble model of decision trees for classification and regression prediction. We demonstrate its utility in genomic selection-assisted breeding with a large dataset of inbred and hybrid maize lines. LightGBM exhibits superior performance in terms of prediction precision, model stability, and computing efficiency through a series of benchmark tests. We also assess the factors that are essential to ensure the best performance of genomic selection prediction by taking complex scenarios in crop hybrid breeding into account. LightGBM has been implemented as a toolbox, CropGBM, encompassing multiple novel functions and analytical modules to facilitate genomically designed breeding in crops.

genetics & heredity,biotechnology & applied microbiology
An experimental evaluation of large scale GBDT systems

FuFangeheng,JiangJiawei,ShaoYingxia,CuiBin

2019-01-01

Abstract:Gradient boosting decision tree (GBDT) is a widely-used machine learning algorithm in both data analytic competitions and real-world industrial applications. Further, driven by the rapid increase i...
The role of LightGBM model in management efficiency enhancement of listed agricultural companies

Xi Xi

DOI: https://doi.org/10.2478/amns.2023.2.00386

2023-09-27

Applied Mathematics and Nonlinear Sciences

Abstract:Abstract This paper first explores the principle of the the LightGBM model, which uses a gradient-based one-sided sampling GOSS procedure to sample the sample data to ensure the accuracy of the procedure in information gain evaluation. The mutually exclusive feature bundling Histogram procedure is used to bind the feature variables of the data and the Leaf-wise strategy for differential splitting to enhance the training speed of the model. Then the steps of the LightGBM procedure are discussed, using a a decision tree as the base learner, then iterative training in sequence, and greedy learning using forward distributed procedure and the original feature fetches must be identified from the bundled features to reduce the training complexity. Finally, the LightGBM procedure is associated and investigated with the other four procedures in terms of discriminative accuracy and discriminative time of the factors affecting the efficiency of enterprise management. In terms of level discrimination accuracy, the level discrimination rate of the LightGBM model is basically above 0.7, and the discrimination rate reaches 0.9 at level 6, which is better than other procedures. The discriminatory time of the LightGBM model is only 28 seconds, which is about 16 times faster than the CART model and 20 times faster than the support vector machine model.
Benchmarking and Optimization of Gradient Boosting Decision Tree Algorithms

Andreea Anghel,Nikolaos Papandreou,Thomas Parnell,Alessandro De Palma,Haralampos Pozidis

DOI: https://doi.org/10.48550/arXiv.1809.04559

2019-01-17

Abstract:Gradient boosting decision trees (GBDTs) have seen widespread adoption in academia, industry and competitive data science due to their state-of-the-art performance in many machine learning tasks. One relative downside to these models is the large number of hyper-parameters that they expose to the end-user. To maximize the predictive power of GBDT models, one must either manually tune the hyper-parameters, or utilize automated techniques such as those based on Bayesian optimization. Both of these approaches are time-consuming since they involve repeatably training the model for different sets of hyper-parameters. A number of software GBDT packages have started to offer GPU acceleration which can help to alleviate this problem. In this paper, we consider three such packages: XGBoost, LightGBM and Catboost. Firstly, we evaluate the performance of the GPU acceleration provided by these packages using large-scale datasets with varying shapes, sparsities and learning tasks. Then, we compare the packages in the context of hyper-parameter optimization, both in terms of how quickly each package converges to a good validation score, and in terms of generalization performance.

Machine Learning
Poster: gbdt-rs: Fast and Trustworthy Gradient Boosting Decision Tree

Tianyi Li,Tongxin Li,Yu Ding,Yulong Zhang,Tao Wei,Xinhui Han

2019-01-01

Abstract:For data analysis services, the capability of preserving user privacy has become an eagerly demanded trait. However, most existing approaches are either unsound to close some major information leakage chanels or prohibitively expensive for practical deployment. We propose gbdt-rs , a secure and fast implementation of the gradient boosting decision tree algorithm, which is widely used in data mining and machine learning tasks. gbdt-rs is designed to fit traditional computing platforms as well as hardware-enforced trusted environments, i.e., Intel SGX secure enclaves. gbdt-rs is coded in the Rust programming language to exterminate memory errors, while its performance is barely sacrificed due to sophisticatedly optimized memory access patterns. Our experiments show that gbdt-rs can provide up to 10x speedup on inference tasks compared to XGBoost, a reputed C++ implementation of the same algorithm. In addition, gbdt-rs is highly auditable due to its relatively small code base.
Unbiased Gradient Boosting Decision Tree with Unbiased Feature Importance

Zheyu Zhang,Tianping Zhang,Jian Li

2023-05-18

Abstract:Gradient Boosting Decision Tree (GBDT) has achieved remarkable success in a wide variety of applications. The split finding algorithm, which determines the tree construction process, is one of the most crucial components of GBDT. However, the split finding algorithm has long been criticized for its bias towards features with a large number of potential splits. This bias introduces severe interpretability and overfitting issues in GBDT. To this end, we provide a fine-grained analysis of bias in GBDT and demonstrate that the bias originates from 1) the systematic bias in the gain estimation of each split and 2) the bias in the split finding algorithm resulting from the use of the same data to evaluate the split improvement and determine the best split. Based on the analysis, we propose unbiased gain, a new unbiased measurement of gain importance using out-of-bag samples. Moreover, we incorporate the unbiased property into the split finding algorithm and develop UnbiasedGBM to solve the overfitting issue of GBDT. We assess the performance of UnbiasedGBM and unbiased gain in a large-scale empirical study comprising 60 datasets and show that: 1) UnbiasedGBM exhibits better performance than popular GBDT implementations such as LightGBM, XGBoost, and Catboost on average on the 60 datasets and 2) unbiased gain achieves better average performance in feature selection than popular feature importance methods. The codes are available at <a class="link-external link-https" href="https://github.com/ZheyuAqaZhang/UnbiasedGBM" rel="external noopener nofollow">this https URL</a>.

Machine Learning
Accelerating Gradient Boosting Machine

Haihao Lu,Sai Praneeth Karimireddy,Natalia Ponomareva,Vahab Mirrokni

DOI: https://doi.org/10.48550/arXiv.1903.08708

2020-08-28

Abstract:Gradient Boosting Machine (GBM) is an extremely powerful supervised learning algorithm that is widely used in practice. GBM routinely features as a leading algorithm in machine learning competitions such as Kaggle and the KDDCup. In this work, we propose Accelerated Gradient Boosting Machine (AGBM) by incorporating Nesterov's acceleration techniques into the design of GBM. The difficulty in accelerating GBM lies in the fact that weak (inexact) learners are commonly used, and therefore the errors can accumulate in the momentum term. To overcome it, we design a "corrected pseudo residual" and fit best weak learner to this corrected pseudo residual, in order to perform the z-update. Thus, we are able to derive novel computational guarantees for AGBM. This is the first GBM type of algorithm with theoretically-justified accelerated convergence rate. Finally we demonstrate with a number of numerical experiments the effectiveness of AGBM over conventional GBM in obtaining a model with good training and/or testing data fidelity.

Machine Learning
Soft Gradient Boosting Machine

Feng Ji,Xu Yi-Xuan,Jiang Yuan,Zhou Zhi-Hua

2020-01-01

Abstract: Gradient Boosting Machine has proven to be one successful function approximator and has been widely used in a variety of areas. However, since the training procedure of each base learner has to take the sequential order, it is infeasible to parallelize the training process among base learners for speed-up. In addition, under online or incremental learning settings, GBMs achieved sub-optimal performance due to the fact that the previously trained base learners can not adapt with the environment once trained. In this work, we propose the soft Gradient Boosting Machine (sGBM) by wiring multiple differentiable base learners together, by injecting both local and global objectives inspired from gradient boosting, all base learners can then be jointly optimized with linear speed-up. When using differentiable soft decision trees as base learner, such device can be regarded as an alternative version of the (hard) gradient boosting decision trees with extra benefits. Experimental results showed that, sGBM enjoys much higher time efficiency with better accuracy, given the same base learner in both on-line and off-line settings.
Multi-grained and multi-layered gradient boosting decision tree for credit scoring

Wan’an Liu,Hong Fan,Min Xia

DOI: https://doi.org/10.1007/s10489-021-02715-6

IF: 5.3

2021-08-10

Applied Intelligence

Abstract:Credit scoring is an important process for banks and financial institutions to manage credit risk. Tree-based ensemble algorithms have made promising progress in credit scoring. However, tree-based ensemble algorithms lack representation learning, making them cannot well express the potential distribution of loan data. In this study, we propose a multi-grained and multi-layered gradient boosting decision tree (GBDT) for credit scoring. Multi-layered GBDT considers the advantages of the explicit learning process of tree-based model and the representation learning ability to discriminate good/bad applicants; multi-grained scanning augments original credit features while enhancing the representation learning ability of multi-layered GBDT. The experimental results on 6 credit scoring datasets show that the hierarchical structure can effectively reduce the intra-class distance and increase the inter-class distance of the credit scoring dataset. In addition, Multi-grained feature augmentation effectively increases the diversity of prediction and further improves the performance of credit scoring, providing more precise credit scoring results.

computer science, artificial intelligence
Predicting and Analyzing Accident Severity with Improved LightGBM Method

Wei Tang,Guoqiang Li,Xiaodong Yang,Dantong Wang

DOI: https://doi.org/10.1109/ICSP58490.2023.10248671

2023-04-21

Abstract:This paper proposes an improved light gradient boosting machine model for traffic accident severity prediction and analysis, which addresses the problems of class imbalance and poor interpretability. The model uses the CU_SMOTE sampling algorithm and the focal loss function using the whale optimization algorithm. The model is validated by real accident data and the factors of fatal accidents are analyzed by Shapley Additive Explanations. The results show that the model outperforms random forest and light gradient boosting machine in accuracy and recall, and that the absence of safety devices, accidents on regular roads, and frontal collisions are the main causes of fatal accidents.

Engineering,Computer Science
LightGBM robust optimization algorithm based on topological data analysis

Han Yang,Guangjun Qin,Ziyuan Liu,Yongqing Hu,Qinglong Dai

2024-06-19

Abstract:To enhance the robustness of the Light Gradient Boosting Machine (LightGBM) algorithm for image classification, a topological data analysis (TDA)-based robustness optimization algorithm for LightGBM, TDA-LightGBM, is proposed to address the interference of noise on image classification. Initially, the method partitions the feature engineering process into two streams: pixel feature stream and topological feature stream for feature extraction respectively. Subsequently, these pixel and topological features are amalgamated into a comprehensive feature vector, serving as the input for LightGBM in image classification tasks. This fusion of features not only encompasses traditional feature engineering methodologies but also harnesses topological structure information to more accurately encapsulate the intrinsic features of the image. The objective is to surmount challenges related to unstable feature extraction and diminished classification accuracy induced by data noise in conventional image processing. Experimental findings substantiate that TDA-LightGBM achieves a 3% accuracy improvement over LightGBM on the SOCOFing dataset across five classification tasks under noisy conditions. In noise-free scenarios, TDA-LightGBM exhibits a 0.5% accuracy enhancement over LightGBM on two classification tasks, achieving a remarkable accuracy of 99.8%. Furthermore, the method elevates the classification accuracy of the Ultrasound Breast Images for Breast Cancer dataset and the Masked CASIA WebFace dataset by 6% and 15%, respectively, surpassing LightGBM in the presence of noise. These empirical results underscore the efficacy of the TDA-LightGBM approach in fortifying the robustness of LightGBM by integrating topological features, thereby augmenting the performance of image classification tasks amidst data perturbations.

Machine Learning
TencentBoost: A Gradient Boosting Tree System with Parameter Server

Jie Jiang,Jiawei Jiang,Bin Cui,Ce Zhang

DOI: https://doi.org/10.1109/icde.2017.87

2017-01-01

Abstract:Gradient boosting tree (GBT), a widely used machine learning algorithm, achieves state-of-the-art performance in academia, industry, and data analytics competitions. Although existing scalable systems which implement GBT, such as XGBoost and MLlib, perform well for datasets with medium-dimensional features, they can suffer performance degradation for many industrial applications where the trained datasets contain highdimensional features. The performance degradation derives from their inefficient mechanisms for model aggregation-either mapreduce or all-reduce. To address this high-dimensional problem, we propose a scalable execution plan using the parameter server architecture to facilitate the model aggregation. Further, we introduce a sparse-pull method and an efficient index structure to increase the processing speed. We implement a GBT system, namely TencentBoost, in the production cluster of Tencent Inc. The empirical results show that our system is 2-20× faster than existing platforms.
LightGBM: An Effective and Scalable Algorithm for Prediction of Chemical Toxicity-Application to the Tox21 and Mutagenicity Data Sets

Jin Zhang,Daniel Mucs,Ulf Norinder,Fredrik Svensson

DOI: https://doi.org/10.1021/acs.jcim.9b00633

2019-10-28

Abstract:Machine learning algorithms have attained widespread use in assessing the potential toxicities of pharmaceuticals and industrial chemicals because of their faster speed and lower cost compared to experimental bioassays. Gradient boosting is an effective algorithm that often achieves high predictivity, but historically the relative long computational time limited its applications in predicting large compound libraries or developing in silico predictive models that require frequent retraining. LightGBM, a recent improvement of the gradient boosting algorithm, inherited its high predictivity but resolved its scalability and long computational time by adopting a leaf-wise tree growth strategy and introducing novel techniques. In this study, we compared the predictive performance and the computational time of LightGBM to deep neural networks, random forests, support vector machines, and XGBoost. All algorithms were rigorously evaluated on publicly available Tox21 and mutagenicity data sets using a Bayesian optimization integrated nested 10-fold cross-validation scheme that performs hyperparameter optimization while examining model generalizability and transferability to new data. The evaluation results demonstrated that LightGBM is an effective and highly scalable algorithm offering the best predictive performance while consuming significantly shorter computational time than the other investigated algorithms across all Tox21 and mutagenicity data sets. We recommend LightGBM for applications of in silico safety assessment and also other areas of cheminformatics to fulfill the ever-growing demand for accurate and rapid prediction of various toxicity or activity related end points of large compound libraries present in the pharmaceutical and chemical industry.
DeepGBM

Guolin Ke,Zhenhui Xu,Jia Zhang,Jiang Bian,Tie‐Yan Liu

DOI: https://doi.org/10.1145/3292500.3330858

2019-01-01

Abstract:Online prediction has become one of the most essential tasks in many real-world applications. Two main characteristics of typical online prediction tasks include tabular input space and online data generation. Specifically, tabular input space indicates the existence of both sparse categorical features and dense numerical ones, while online data generation implies continuous task-generated data with potentially dynamic distribution. Consequently, effective learning with tabular input space as well as fast adaption to online data generation become two vital challenges for obtaining the online prediction model. Although Gradient Boosting Decision Tree (GBDT) and Neural Network (NN) have been widely used in practice, either of them yields their own weaknesses. Particularly, GBDT can hardly be adapted to dynamic online data generation, and it tends to be ineffective when facing sparse categorical features; NN, on the other hand, is quite difficult to achieve satisfactory performance when facing dense numerical features. In this paper, we propose a new learning framework, DeepGBM, which integrates the advantages of the both NN and GBDT by using two corresponding NN components: (1) CatNN, focusing on handling sparse categorical features. (2) GBDT2NN, focusing on dense numerical features with distilled knowledge from GBDT. Powered by these two components, DeepGBM can leverage both categorical and numerical features while retaining the ability of efficient online update. Comprehensive experiments on a variety of publicly available datasets have demonstrated that DeepGBM can outperform other well-recognized baselines in various online prediction tasks.
GBDT-MO: Gradient Boosted Decision Trees for Multiple Outputs

Zhendong Zhang,Cheolkon Jung

DOI: https://doi.org/10.48550/arXiv.1909.04373

2019-12-28

Abstract:Gradient boosted decision trees (GBDTs) are widely used in machine learning, and the output of current GBDT implementations is a single variable. When there are multiple outputs, GBDT constructs multiple trees corresponding to the output variables. The correlations between variables are ignored by such a strategy causing redundancy of the learned tree structures. In this paper, we propose a general method to learn GBDT for multiple outputs, called GBDT-MO. Each leaf of GBDT-MO constructs predictions of all variables or a subset of automatically selected variables. This is achieved by considering the summation of objective gains over all output variables. Moreover, we extend histogram approximation into multiple output case to speed up the training process. Various experiments on synthetic and real-world datasets verify that GBDT-MO achieves outstanding performance in terms of both accuracy and training speed. Our codes are available on-line.

Computer Vision and Pattern Recognition,Machine Learning

990 LightGBM: A highly efficient gradient boosting decision tree

DimBoost

Gradient Boosting With Piece-Wise Linear Regression Trees

Quantized Training of Gradient Boosting Decision Trees

MT-GBM: A Multi-Task Gradient Boosting Machine with Shared Decision Trees

HarpGBDT: Optimizing Gradient Boosting Decision Tree for Parallel Efficiency

LightGBM: accelerated genomically designed crop breeding through ensemble learning

An experimental evaluation of large scale GBDT systems

The role of LightGBM model in management efficiency enhancement of listed agricultural companies

Benchmarking and Optimization of Gradient Boosting Decision Tree Algorithms

Poster: gbdt-rs: Fast and Trustworthy Gradient Boosting Decision Tree

Unbiased Gradient Boosting Decision Tree with Unbiased Feature Importance

Accelerating Gradient Boosting Machine

Soft Gradient Boosting Machine

Multi-grained and multi-layered gradient boosting decision tree for credit scoring

Predicting and Analyzing Accident Severity with Improved LightGBM Method

LightGBM robust optimization algorithm based on topological data analysis

TencentBoost: A Gradient Boosting Tree System with Parameter Server

LightGBM: An Effective and Scalable Algorithm for Prediction of Chemical Toxicity-Application to the Tox21 and Mutagenicity Data Sets

DeepGBM

GBDT-MO: Gradient Boosted Decision Trees for Multiple Outputs