Abstract:Generalized linear models are widely used in data analysis and machine learning, especially in large-scale machine learning because of its simplicity and good performance. Generalized linear models include regression, like linear regression, lasso and classification, support vector machine and logistic regression. We have some popular optimization methods to solve them, including stochastic gradient descent, coordinate descent and alternating direction method of multipliers. Commonly used systems for generalized linear model use a single optimization algorithm to solve all kinds of models. However, experiments show that it is impossible to achieve a cost-effective solution for all kinds of generalized linear models due to the differences between different models. In order to resolve the problem, we propose a rule-based optimization algorithm selector to select the best optimization algorithm automatically. In this paper, we first broadly review the three commonly used optimization methods, stochastic gradient, coordinate descent, alternating direction method of multipliers, perform some experiments, and then propose a rule-based optimizer to guide the solving. We also design a new system, GLM+, based on the three rules we proposed with the two engineering techniques, parameter selection and optimization for sparse data. We conduct some experiments on real datasets and compare it with the commonly used machine learning systems, Shotgun and scikit-learn, to verify the effectiveness of our system GLM+. The experiments demonstrate that GLM+ can be 3-10×, sometimes 20×, faster than existing popular systems for generalized linear models.

LinXGBoost: Extension of XGBoost to Generalized Local Linear Models

Local Linear M‐estimation in Non‐parametric Spatial Regression

Generalized XGBoost Method

A Local Online Learning Approach for Non-linear Data.

Why (and When) does Local SGD Generalize Better than SGD?

Multi-Target XGBoostLSS Regression

GLM+: An Efficient System for Generalized Linear Models

Continuously Generalized Ordinal Regression for Linear and Deep Models

A comparative analysis of gradient boosting algorithms

Stability and Generalization for Minibatch SGD and Local SGD

XGBoostLSS -- An extension of XGBoost to probabilistic forecasting

The Behavior and Convergence of Local Bayesian Optimization

Survival regression with accelerated failure time model in XGBoost

Logistic Regression, AdaBoost and Bregman Distances

Generalised Boosted Forests

Local Linear Forests

FairXGBoost: Fairness-aware Classification in XGBoost

Optimization by gradient boosting

Scaling Up Diffusion and Flow-based XGBoost Models

G-LIME: Statistical Learning for Local Interpretations of Deep Neural Networks Using Global Priors.

Learning Nonlinear Functions Using Regularized Greedy Forest