Abstract:Recent years have witnessed significant success in Gradient Boosting Decision Trees (GBDT) for a wide range of machine learning applications. Generally, a consensus about GBDT's training algorithms is gradients and statistics are computed based on high-precision floating points. In this paper, we investigate an essentially important question which has been largely ignored by the previous literature: how many bits are needed for representing gradients in training GBDT? To solve this mystery, we propose to quantize all the high-precision gradients in a very simple yet effective way in the GBDT's training algorithm. Surprisingly, both our theoretical analysis and empirical studies show that the necessary precisions of gradients without hurting any performance can be quite low, e.g., 2 or 3 bits. With low-precision gradients, most arithmetic operations in GBDT training can be replaced by integer operations of 8, 16, or 32 bits. Promisingly, these findings may pave the way for much more efficient training of GBDT from several aspects: (1) speeding up the computation of gradient statistics in histograms; (2) compressing the communication cost of high-precision statistical information during distributed training; (3) the inspiration of utilization and development of hardware architectures which well support low-precision computation for GBDT training. Benchmarked on CPUs, GPUs, and distributed clusters, we observe up to 2$\times$ speedup of our simple quantization strategy compared with SOTA GBDT systems on extensive datasets, demonstrating the effectiveness and potential of the low-precision training of GBDT. The code will be released to the official repository of LightGBM.

Asynch-SGBDT: Train Stochastic Gradient Boosting Decision Trees in an Asynchronous Parallel Manner.

HarpGBDT: Optimizing Gradient Boosting Decision Tree for Parallel Efficiency

WBSP: Addressing Stragglers in Distributed Machine Learning with Worker-Busy Synchronous Parallel

Challenges and Opportunities of Building Fast GBDT Systems.

Asynchronous Accelerated Stochastic Gradient Descent.

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

Fast Asynchronous Parallel Stochastic Gradient Decent

Gradient Boosting With Piece-Wise Linear Regression Trees

Quantized Training of Gradient Boosting Decision Trees

Optimization of Parallel Stochastic Gradient Descent on Sunway TaihuLight

Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization

Benchmarking and Optimization of Gradient Boosting Decision Tree Algorithms

SAP-SGD: Accelerating Distributed Parallel Training with High Communication Efficiency on Heterogeneous Clusters

Parallel Training GBRT Based on KMeans Histogram Approximation for Big Data.

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Asynchronous Local-SGD Training for Language Modeling

Dual-Delayed Asynchronous SGD for Arbitrarily Heterogeneous Data

MT-GBM: A Multi-Task Gradient Boosting Machine with Shared Decision Trees

Make Workers Work Harder: Decoupled Asynchronous Proximal Stochastic Gradient Descent.

Asynchronous Stochastic Gradient Descent with Decoupled Backpropagation and Layer-Wise Updates

Guided parallelized stochastic gradient descent for delay compensation