Abstract:Forward stagewise regression is a simple algorithm that can be used to estimate regularized models. The updating rule adds a small constant to a regression coefficient in each iteration, such that the underlying optimization problem is solved slowly with small improvements. This is similar to gradient boosting, with the essential difference that the step size is determined by the product of the gradient and a step length parameter in the latter algorithm. One often overlooked challenge in gradient boosting for distributional regression is the issue of a vanishing small gradient, which practically halts the algorithm's progress. We show that gradient boosting in this case oftentimes results in suboptimal models, especially for complex problems certain distributional parameters are never updated due to the vanishing gradient. Therefore, we propose a stagewise boosting-type algorithm for distributional regression, combining stagewise regression ideas with gradient boosting. Additionally, we extend it with a novel regularization method, correlation filtering, to provide additional stability when the problem involves a large number of covariates. Furthermore, the algorithm includes best-subset selection for parameters and can be applied to big data problems by leveraging stochastic approximations of the updating steps. Besides the advantage of processing large datasets, the stochastic nature of the approximations can lead to better results, especially for complex distributions, by reducing the risk of being trapped in a local optimum. The performance of our proposed stagewise boosting distributional regression approach is investigated in an extensive simulation study and by estimating a full probabilistic model for lightning counts with data of more than 9.1 million observations and 672 covariates.

Feature Extraction in Signal Regression: A Boosting Technique for Functional Data Regression

Boosting the Partial Least Square Algorithm for Regression Modelling

Boosting The Pls Algorithm For Regressive Modelling

Tree-based boosting with functional data

Extrema-weighted feature extraction for functional data

Boosting Functional Regression Models with FDboost

MODIFIED KERNEL-BASED NONLINEAR FEATURE EXTRACTION

Robust Functional Regression with Discretely Sampled Predictors

Robust Function-on-Function Regression

A regression model with a hidden logistic process for feature extraction from time series

A new computationally efficient algorithm to solve Feature Selection for Functional Data Classification in high-dimensional spaces

Robust boosting for regression problems

Boosted-SpringDTW for Comprehensive Feature Extraction of PPG Signals

Boosted-SpringDTW for Comprehensive Feature Extraction of Physiological Signals

Functional Linear Regression with Mixed Predictors

Supervised Principal Component Regression for Functional Responses with High Dimensional Predictors

Extending Statistical Boosting - An Overview of Recent Methodological Developments

Stagewise Boosting Distributional Regression

Goal-oriented Feature Extraction: a novel approach for enhancing data-driven surrogate model

Functional Linear Regression for Discretely Observed Data: from Ideal to Reality

Boosting Factor-Specific Functional Historical Models for the Detection of Synchronisation in Bioelectrical Signals