Abstract:In data mining, regression analysis is a computational tool that predicts continuous output variables from a number of independent input variables, by approximating their complex inner relationship. A large number of methods have been successfully proposed, based on various methodologies, including linear regression, support vector regression, neural network, piece-wise regression, etc. In terms of piece-wise regression, the existing methods in literature are usually restricted to problems of very small scale, due to their inherent non-linear nature. In this work, a more efficient piece-wise linear regression method is introduced based on a novel integer linear programming formulation. The proposed method partitions one input variable into multiple mutually exclusive segments, and fits one multivariate linear regression function per segment to minimise the total absolute error. Assuming both the single partition feature and the number of regions are known, the mixed integer linear model is proposed to simultaneously determine the locations of multiple break-points and regression coefficients for each segment. Furthermore, an efficient heuristic procedure is presented to identify the key partition feature and final number of break-points. 7 real world problems covering several application domains have been used to demonstrate the efficiency of our proposed method. It is shown that our proposed piece-wise regression method can be solved to global optimality for datasets of thousands samples, which also consistently achieves higher prediction accuracy than a number of state-of-the-art regression methods. Another advantage of the proposed method is that the learned model can be conveniently expressed as a small number of if-then rules that are easily interpretable. Overall, this work proposes an efficient rule-based multivariate regression method based on piece-wise functions and achieves better prediction performance than state-of-the-arts approaches. This novel method can benefit expert systems in various applications by automatically acquiring knowledge from databases to improve the quality of knowledge base.

A mathematical programming approach for integrated multiple linear regression subset selection and validation

Subset selection for multiple linear regression via optimization

Mathematical programming for piecewise linear regression analysis

An efficient optimization approach for best subset selection in linear regression, with application to model selection and fitting in autoregressive time-series

An automated exact solution framework towards solving the logistic regression best subset selection problem

A Discrete Choice Model for Subset Selection

Best Subset Solution Path for Linear Dimension Reduction Models using Continuous Optimization

Selecting an Optimal Subset with Regression Metamodels

Multi-Model Subset Selection

Budget constrained model selection for multiple linear regression

A Novel Variable Selection Approach for Redundant Information Elimination Purpose of Process Control

A one-covariate-at-a-time multiple testing approach to variable selection in additive models

Metamodel selection based on stepwise regression

A Unified Variable Selection Approach for Varying Coefficient Models

A robust and efficient estimation and variable selection method for partially linear models with large-dimensional covariates

Iteratively variable subset optimization for multivariate calibration

Bayesian subset selection and variable importance for interpretable prediction and classification

Best Subset Selection via a Modern Optimization Lens

Subdata selection algorithm for linear model discrimination

Penalized regression with multiple loss functions and selection by vote

Validation of Linear Programming Models