Abstract:We present a novel method for exact hierarchical sparse polynomial regression. Our regressor is that degree $r$ polynomial which depends on at most $k$ inputs, counting at most $\ell$ monomial terms, which minimizes the sum of the squares of its prediction errors. The previous hierarchical sparse specification aligns well with modern big data settings where many inputs are not relevant for prediction purposes and the functional complexity of the regressor needs to be controlled as to avoid overfitting. We present a two-step approach to this hierarchical sparse regression problem. First, we discard irrelevant inputs using an extremely fast input ranking heuristic. Secondly, we take advantage of modern cutting plane methods for integer optimization to solve our resulting reduced hierarchical $(k, \ell)$-sparse problem exactly. The ability of our method to identify all $k$ relevant inputs and all $\ell$ monomial terms is shown empirically to experience a phase transition. Crucially, the same transition also presents itself in our ability to reject all irrelevant features and monomials as well. In the regime where our method is statistically powerful, its computational complexity is interestingly on par with Lasso based heuristics. The presented work fills a void in terms of a lack of powerful disciplined nonlinear sparse regression methods in high-dimensional settings. Our method is shown empirically to scale to regression problems with $n\approx 10,000$ observations for input dimension $p\approx 1,000$.

Polynomial regression modeling based on Gram-Schmidt process

Multivariable Fractional Polynomial Method for Regression Model.

Two-dimensional recursive least squares identification based on local polynomial modeling for batch processes.

Recursive Least-Squares Method with Membership Functions

ADDITIVE COEFFICIENT MODELING VIA POLYNOMIAL SPLINE

A NPLS modeling method of outer-inner polynomial model based on variable selection

Nonlinear Regression Automatic Modeling Process

Polynomial spline estimation for partial functional linear regression models

Model selection of polynomial kernel regression

Variable selection in discriminant analysis based on Gram-Schmidt process

Dimension Reduction Method of Support Vector Machine Based on Gram-Schmidt Process

A sparse interpolation algorithm based on modular arithmetic coefficient parsing

NONLINEAR STRUCTURE ANALYSIS WITH PARTIAL LEAST-SQUARES REGRESSION BASED ON SPLINE TRANSFORMATION

Nonlinear Recursive Partial Least Squares and Its Applications

Modeling of batch process based on higher order partial least squares

Towards a mathematical framework to inform neural network modelling via polynomial regression

Sparse Hierarchical Regression with Polynomials

Gram-Schmidt regression and application in cutting tool abrasion prediction

Functional variable selection via Gram–Schmidt orthogonalization for multiple functional linear regression

POLYNOMIAL MODEL BASED ON LINEAR TRANSFORMATION

Second order hierarchical partial least squares regression-polynomial chaos expansion for global sensitivity and reliability analyses of high-dimensional models