Abstract:Forward stagewise regression is a simple algorithm that can be used to estimate regularized models. The updating rule adds a small constant to a regression coefficient in each iteration, such that the underlying optimization problem is solved slowly with small improvements. This is similar to gradient boosting, with the essential difference that the step size is determined by the product of the gradient and a step length parameter in the latter algorithm. One often overlooked challenge in gradient boosting for distributional regression is the issue of a vanishing small gradient, which practically halts the algorithm's progress. We show that gradient boosting in this case oftentimes results in suboptimal models, especially for complex problems certain distributional parameters are never updated due to the vanishing gradient. Therefore, we propose a stagewise boosting-type algorithm for distributional regression, combining stagewise regression ideas with gradient boosting. Additionally, we extend it with a novel regularization method, correlation filtering, to provide additional stability when the problem involves a large number of covariates. Furthermore, the algorithm includes best-subset selection for parameters and can be applied to big data problems by leveraging stochastic approximations of the updating steps. Besides the advantage of processing large datasets, the stochastic nature of the approximations can lead to better results, especially for complex distributions, by reducing the risk of being trapped in a local optimum. The performance of our proposed stagewise boosting distributional regression approach is investigated in an extensive simulation study and by estimating a full probabilistic model for lightning counts with data of more than 9.1 million observations and 672 covariates.

Scalable Estimation for Structured Additive Distributional Regression

Stagewise Boosting Distributional Regression

Online Distributional Regression

Boosting Multivariate Structured Additive Distributional Regression Models

Stochastic gradient descent methods for estimation with large data sets

Distributing the Stochastic Gradient Sampler for Large-Scale LDA.

Distributed optimization and statistical learning for large-scale penalized expectile regression

Large-Scale Distributed Bayesian Matrix Factorization using Stochastic Gradient MCMC

Robust Data-driven Sparse Estimation of Distribution Factors Considering PMU Data Quality and Renewable Energy Uncertainty - Part II: Scalability and Applications

Scalable Estimation and Inference with Large-scale or Online Survival Data

Provably Efficient Posterior Sampling for Sparse Linear Regression via Measure Decomposition

Distribution-Free Location-Scale Regression

Distributionally Robust Optimization as a Scalable Framework to Characterize Extreme Value Distributions

High-Dimensional Distributed Sparse Classification with Scalable Communication-Efficient Global Updates

A new large-scale learning algorithm for generalized additive models

Two-step estimation of high dimensional additive models

Distributional Adaptive Soft Regression Trees

Group Selection and Shrinkage: Structured Sparsity for Semiparametric Additive Models

Least Squares Approximation for a Distributed System

Relative error-based distributed estimation in growing dimensions

Scalable visualisation methods for modern Generalized Additive Models