Comparison of prediction methods for multicollinear data

Tormod Naes,Harald Martens

DOI: https://doi.org/10.1080/03610918508812458

1985-01-01

Abstract:In this paper we discuss the partial least squares (PLS) prediction method. The method is compared to the predictor based on principal component regression (PCR). Both theoretical considerations and computations on artificial and real data are presented.

statistics & probability

What problem does this paper attempt to address?

A Two-Step PLS Inspired Method for Linear Prediction with Group Effect

Ying Li,Peter Udén,Dietrich von Rosen

DOI: https://doi.org/10.1007/s13171-012-0022-8

2015-01-01

Sankhya A

Abstract:In this article, we consider prediction of a univariate response from background data. The data may have a near-collinear structure and additionally group effects are assumed to exist. A two-step method is proposed. The first step summarizes the information in the predictors via a bilinear model. The bilinear model has a Krylov structured within individual design matrix, which is the link to classical partial least squares (PLS) analysis and a between-individual design matrix which handles group effects. The second step is the prediction step where a conditional expectation approach is used. The two-step method gives new insight into PLS. Explicit maximum likelihood estimators of the dispersion matrix and mean for the predictors are derived under the assumption that the covariance between the response and explanatory variables is known. It is shown that for within-sample prediction the mean squared error of the two-step method is always smaller than PLS.
Maximum Likelihood Estimators in a Two Step Model for PLS

Ying Li,Dietrich von Rosen

DOI: https://doi.org/10.1080/03610926.2011.607531

2012-01-01

Abstract:Univariate partial least squares regression (PLS1) is a method of modeling relationships between a response variable and explanatory variables, especially when the explanatory variables are almost collinear. The purpose is to predict a future response observation, although in many applications there is an interest to understand the contributions of each explanatory variable. It is an algorithmic approach. In this article, we are going to use the algorithm presented by Helland (1988). The population PLS predictor is linked to a linear model including a Krylov design matrix and a two-step estimation procedure. For the first step, the maximum likelihood approach is applied to a specific multivariate linear model, generating tools for evaluating the information in the explanatory variables. It is shown that explicit maximum likelihood estimators of the dispersion matrix can be obtained where the dispersion matrix, besides representing the variation in the error, also includes the Krylov structured design matrix describing the mean.
Comparing methods addressing multi-collinearity when developing prediction models

Artuur M. Leeuwenberg,Maarten van Smeden,Johannes A. Langendijk,Arjen van der Schaaf,Murielle E. Mauer,Karel G.M. Moons,Johannes B. Reitsma,Ewoud Schuit

DOI: https://doi.org/10.48550/arXiv.2101.01603

2021-01-05

Abstract:Clinical prediction models are developed widely across medical disciplines. When predictors in such models are highly collinear, unexpected or spurious predictor-outcome associations may occur, thereby potentially reducing face-validity and explainability of the prediction model. Collinearity can be dealt with by exclusion of collinear predictors, but when there is no a priori motivation (besides collinearity) to include or exclude specific predictors, such an approach is arbitrary and possibly inappropriate. We compare different methods to address collinearity, including shrinkage, dimensionality reduction, and constrained optimization. The effectiveness of these methods is illustrated via simulations. In the conducted simulations, no effect of collinearity was observed on predictive outcomes. However, a negative effect of collinearity on the stability of predictor selection was found, affecting all compared methods, but in particular methods that perform strong predictor selection (e.g., Lasso).}

Methodology
Prediction of multivariate responses with a select number of principal components

Inge Koch,Kanta Naito

DOI: https://doi.org/10.48550/arXiv.0807.4010

2008-07-25

Methodology

Abstract:This paper proposes a new method and algorithm for predicting multivariate responses in a regression setting. Research into classification of High Dimension Low Sample Size (HDLSS) data, in particular microarray data, has made considerable advances, but regression prediction for high-dimensional data with continuous responses has had less attention. Recently Bair et al (2006) proposed an efficient prediction method based on supervised principal component regression (PCR). Motivated by the fact that a larger number of principal components results in better regression performance, this paper extends the method of Bair et al in several ways: a comprehensive variable ranking is combined with a selection of the best number of components for PCR, and the new method further extends to regression with multivariate responses. The new method is particularly suited to HDLSS problems. Applications to simulated and real data demonstrate the performance of the new method. Comparisons with Bair et al (2006) show that for high-dimensional data in particular the new ranking results in a smaller number of predictors and smaller errors.
Power Analysis of Principal Components Regression in Genetic Association Studies.

Yan-feng Shen,Jun Zhu

DOI: https://doi.org/10.1631/jzus.b0830866

2009-01-01

Journal of Zhejiang University SCIENCE B

Abstract:Association analysis provides an opportunity to find genetic variants underlying complex traits. A principal components regression (PCR)-based approach was shown to outperform some competing approaches. However, a limitation of this method is that the principal components (PCs) selected from single nucleotide polymorphisms (SNPs) may be unrelated to the phenotype. In this article, we investigate the theoretical properties of such a method in more detail. We first derive the exact power function of the test based on PCR, and hence clarify the relationship between the test power and the degrees of freedom (DF). Next, we extend the PCR test to a general weighted PCs test, which provides a unified framework for understanding the properties of some related statistics. We then compare the performance of these tests. We also introduce several data-driven adaptive alternatives to overcome difficulties in the PCR approach. Finally, we illustrate our results using simulations based on real genotype data. Simulation study shows the risk of using the unsupervised rule to determine the number of PCs, and demonstrates that there is no single uniformly powerful method for detecting genetic variants.
Performance of binary prediction models in high-correlation low-dimensional settings: a comparison of methods

Artuur M. Leeuwenberg,Maarten van Smeden,Johannes A. Langendijk,Arjen van der Schaaf,Murielle E. Mauer,Karel G. M. Moons,Johannes B. Reitsma,Ewoud Schuit

DOI: https://doi.org/10.1186/s41512-021-00115-5

2022-01-11

Diagnostic and Prognostic Research

Abstract:Abstract Background Clinical prediction models are developed widely across medical disciplines. When predictors in such models are highly collinear, unexpected or spurious predictor-outcome associations may occur, thereby potentially reducing face-validity of the prediction model. Collinearity can be dealt with by exclusion of collinear predictors, but when there is no a priori motivation (besides collinearity) to include or exclude specific predictors, such an approach is arbitrary and possibly inappropriate. Methods We compare different methods to address collinearity, including shrinkage, dimensionality reduction, and constrained optimization. The effectiveness of these methods is illustrated via simulations. Results In the conducted simulations, no effect of collinearity was observed on predictive outcomes (AUC, R 2 , Intercept, Slope) across methods. However, a negative effect of collinearity on the stability of predictor selection was found, affecting all compared methods, but in particular methods that perform strong predictor selection (e.g., Lasso). Methods for which the included set of predictors remained most stable under increased collinearity were Ridge, PCLR, LAELR, and Dropout. Conclusions Based on the results, we would recommend refraining from data-driven predictor selection approaches in the presence of high collinearity, because of the increased instability of predictor selection, even in relatively high events-per-variable settings. The selection of certain predictors over others may disproportionally give the impression that included predictors have a stronger association with the outcome than excluded predictors.

English Else
Comparison of Canonical Correlation and Partial Least Squares analyses of simulated and empirical data

Anthony R McIntosh

DOI: https://doi.org/10.48550/arXiv.2107.06867

2022-06-27

Abstract:In this paper, we compared the general forms of CCA and PLS on three simulated and two empirical datasets, all having large sample sizes. We took successively smaller subsamples of these data to evaluate sensitivity, reliability, and reproducibility. In null data having no correlation within or between blocks, both methods showed equivalent false positive rates across sample sizes. Both methods also showed equivalent detection in data with weak but reliable effects until sample sizes drop below n=50. In the case of strong effects, both methods showed similar performance unless the correlations of items within one data block were high. For PLS, the results were reproducible across sample sizes for strong effects, except at the smallest sample sizes. On the contrary, the reproducibility for CCA declined when the within-block correlations were high. This was ameliorated if a principal components analysis (PCA) was performed and the component scores used to calculate the cross-block matrix. The outcome of our examination gives three messages. First, for data with reasonable within and between block structure, CCA and PLS give comparable results. Second, if there are high correlations within either block, this can compromise the reliability of CCA results. This known issue of CCA can be remedied with PCA before cross-block calculation. This, however, assumes that the PCA structure is stable for a given sample. Third, null hypothesis testing does not guarantee that the results are reproducible, even with large sample sizes. This final outcome suggests that both statistical significance and reproducibility be assessed for any data.

Methodology,Applications
Partial Least Squares Methods: Partial Least Squares Correlation and Partial Least Square Regression

Hervé Abdi,Lynne J. Williams

DOI: https://doi.org/10.1007/978-1-62703-059-5_23

2012-08-18

Abstract:Partial least square (pls) methods (also sometimes called projection to latent structures) relate the information present in two data tables that collect measurements on the same set of observations. pls methods proceed by deriving latent variables which are (optimal) linear combinations of the variables of a data table. When the goal is to find the shared information between two tables, the approach is equivalent to a correlation problem and the technique is then called partial least square correlation (PLSC) (also sometimes called pls-SVD). In this case there are two sets of latent variables (one set per table), and these latent variables are required to have maximal covariance. When the goal is to predict one data table the other one, the technique is then called partial least square regression. In this case there is one set of latent variables (derived from the predictor table) and these latent variables are required to give the best possible prediction. In this paper we present and illustrate PLSC and PLSR and show how these descriptive multivariate analysis techniques can be extended to deal with inferential questions by using cross-validation techniques such as the bootstrap and permutation tests.
Improvement of simultaneous prediction using principal components approach

Gülesen Üstündağ Şiray

DOI: https://doi.org/10.1080/03610918.2024.2408620

2024-10-14

Communications in Statistics - Simulation and Computation

Abstract:Prediction has a fundamental role both in linear regression analysis and in many applied sciences. The predictions of actual values or mean values of the response variable were obtained separately, earlier than defined the target function. It has been attracting a lot of interest because obtaining simultaneous predictions both of the mean values and actual values is more reasonable in many applied studies since the target function was reported. This paper purposes to improve the simultaneous prediction with the target function when the problem of multicollinearity exists in the linear regression model by defining new predictors based on the principal components estimator, which is more resistant to multicollinearity than the ordinary least squares estimator. The comparisons of proposed predictors are done theoretically by using the mean squared error of predictions. Then theoretical results are supported by an exhaustive simulation study. Also, performance comparisons of the newly suggested predictors are made for two real-life data sets, real estate valuation and cruise ship info. The findings show that the use of the proposed predictors will make a crucial contribution to applied studies in obtaining better predictions.

statistics & probability
Principal component regression in NIR analysis: Viewpoints, background details and selection of components

Tormod Næs,Harald Martens

DOI: https://doi.org/10.1002/cem.1180020207

IF: 2.5

1988-04-01

Journal of Chemometrics

Abstract:In this paper we present formulae for prediction error related to principal component regression (PCR). The difference between PCR and ordinary least‐squares (LS) regression is discussed in relation to these formulae. This discussion is used as a basis for a treatment of PCR in NIR analysis. The theory is illustrated by two examples from NIR analysis.

chemistry, analytical,instruments & instrumentation,mathematics, interdisciplinary applications,automation & control systems,computer science, artificial intelligence,statistics & probability
Principal Model Analysis Based on Partial Least Squares

Qiwei Xie,Liang Tang,Weifu Li,Vijay John,Yong Hu

DOI: https://doi.org/10.48550/arXiv.1902.02422

IF: 5.414

2019-02-06

Machine Learning

Abstract:Motivated by the Bagging Partial Least Squares (PLS) and Principal Component Analysis (PCA) algorithms, we propose a Principal Model Analysis (PMA) method in this paper. In the proposed PMA algorithm, the PCA and the PLS are combined. In the method, multiple PLS models are trained on sub-training sets, derived from the original training set based on the random sampling with replacement method. The regression coefficients of all the sub-PLS models are fused in a joint regression coefficient matrix. The final projection direction is then estimated by performing the PCA on the joint regression coefficient matrix. The proposed PMA method is compared with other traditional dimension reduction methods, such as PLS, Bagging PLS, Linear discriminant analysis (LDA) and PLS-LDA. Experimental results on six public datasets show that our proposed method can achieve better classification performance and is usually more stable.
A correlation principal component regression analysis of NIR data

Jianguo Sun

DOI: https://doi.org/10.1002/cem.1180090104

IF: 2.5

1995-01-01

Journal of Chemometrics

Abstract:The use of principal component regression (PCR) as a multivariate calibration method has been discussed by a number of authors. In most situations principal components are included in the regression model in sequence based on the variances of the components, and the principal components with small variances are rarely used in regression. As pointed out by some authors, a low variance for a component does not necessarily imply that the corresponding component is unimportant, especially when prediction is of primary interest. In this paper we investigate a different version of PCR, correlation principal component regression (CPCR). In CPCR the importance of principal components in terms of predicting the response variable is used as a basis for the inclusion of principal components in the regression model. Two typical examples arising from calibrating near‐infrared (NIR) instruments are discussed for the comparison of the two different versions of PCR along with partial least squares (PLS), a commonly used regression approach in NIR analysis. In both examples the three methods show similar optimal prediction ability, but CPCR performs better than standard PCR and PLS in terms of the number of components needed to achieve the optimal prediction ability. Similar results are also seen in other NIR examples.

chemistry, analytical,instruments & instrumentation,mathematics, interdisciplinary applications,automation & control systems,computer science, artificial intelligence,statistics & probability
Partial least squares regression as an alternative to current regression methods used in ecology

Luis M. Carrascal,Ismael Galván,Oscar Gordo

DOI: https://doi.org/10.1111/j.1600-0706.2008.16881.x

2009-05-01

Oikos

Abstract:This paper briefly presents the aims, requirements and results of partial least squares regression analysis (PLSR), and its potential utility in ecological studies. This statistical technique is particularly well suited to analyzing a large array of related predictor variables (i.e. not truly independent), with a sample size not large enough compared to the number of independent variables, and in cases in which an attempt is made to approach complex phenomena or syndromes that must be defined as a combination of several variables obtained independently. A simulation experiment is carried out to compare this technique with multiple regression (MR) and with a combination of principal component analysis and multiple regression (PCA+MR), varying the number of predictor variables and sample sizes. PLSR models explained a similar amount of variance to those results obtained by MR and PCA+MR. However, PLSR was more reliable than other techniques when identifying relevant variables and their magnitudes of influence, especially in cases of small sample size and low tolerance. Finally, we present one example of PLSR to illustrate its application and interpretation in ecology.

ecology
Structured Principal Component Analysis Model With Variable Correlation Constraint

Ruikun Zhai,Jiusun Zeng,Zhiqiang Ge

DOI: https://doi.org/10.1109/TCST.2021.3069539

IF: 4.8

2022-01-01

IEEE Transactions on Control Systems Technology

Abstract:Principal Component Analysis (PCA) is a widely used technique in process monitoring, fault diagnosis, and soft sensing of industrial systems. Despite its popularity, PCA suffers from the limitations of poor interpretability and robustness. In order to improve the PCA model, this article considers a structured sparse method--Laplace sparse principal component analysis (LSPCA), by integrating domain knowledge and sparsity constraint in the form of Laplace matrix and elastic net regularization constraint. Different from previous work, this article focuses on the advantages brought by the Laplace matrix and sparsity constraint on revealing the true latent process structure. When variable correlations are known in advance, the Laplace matrix can help extract sparse principal components that are consistent with the variable correlations. The increased accuracy of extracted latent process structure enhances the capability of PCA in soft sensing. The performance of the proposed method is verified by a simulation example and an application study to a distillation process.
Principal Components Analysis in Clinical Studies

Zhongheng Zhang,Adela Castello

DOI: https://doi.org/10.21037/atm.2017.07.12

IF: 3.616

2017-01-01

Annals of Translational Medicine

Abstract:In multivariate analysis, independent variables are usually correlated to each other which can introduce multicollinearity in the regression models. One approach to solve this problem is to apply principal components analysis (PCA) over these variables. This method uses orthogonal transformation to represent sets of potentially correlated variables with principal components (PC) that are linearly uncorrelated. PCs are ordered so that the first PC has the largest possible variance and only some components are selected to represent the correlated variables. As a result, the dimension of the variable space is reduced. This tutorial illustrates how to perform PCA in R environment, the example is a simulated dataset in which two PCs are responsible for the majority of the variance in the data. Furthermore, the visualization of PCA is highlighted.
Comparison of variants of canonical correlation analysis and partial least squares for combined analysis of MRI and genetic data

Claudia Grellmann,Sebastian Bitzer,Jane Neumann,Lars T Westlye,Ole A Andreassen,Arno Villringer,Annette Horstmann

DOI: https://doi.org/10.1016/j.neuroimage.2014.12.025

IF: 5.7

2015-02-15

NeuroImage

Abstract:The standard analysis approach in neuroimaging genetics studies is the mass-univariate linear modeling (MULM) approach. From a statistical view, however, this approach is disadvantageous, as it is computationally intensive, cannot account for complex multivariate relationships, and has to be corrected for multiple testing. In contrast, multivariate methods offer the opportunity to include combined information from multiple variants to discover meaningful associations between genetic and brain imaging data. We assessed three multivariate techniques, partial least squares correlation (PLSC), sparse canonical correlation analysis (sparse CCA) and Bayesian inter-battery factor analysis (Bayesian IBFA), with respect to their ability to detect multivariate genotype-phenotype associations. Our goal was to systematically compare these three approaches with respect to their performance and to assess their suitability for high-dimensional and multi-collinearly dependent data as is the case in neuroimaging genetics studies. In a series of simulations using both linearly independent and multi-collinear data, we show that sparse CCA and PLSC are suitable even for very high-dimensional collinear imaging data sets. Among those two, the predictive power was higher for sparse CCA when voxel numbers were below 400 times sample size and candidate SNPs were considered. Accordingly, we recommend Sparse CCA for candidate phenotype, candidate SNP studies. When voxel numbers exceeded 500 times sample size, the predictive power was the highest for PLSC. Therefore, PLSC can be considered a promising technique for multivariate modeling of high-dimensional brain-SNP-associations. In contrast, Bayesian IBFA cannot be recommended, since additional post-processing steps were necessary to detect causal relations. To verify the applicability of sparse CCA and PLSC, we applied them to an experimental imaging genetics data set provided for us. Most importantly, application of both methods replicated the findings of this data set.
Multivariate Multilinear Regression

Ya Su,Xinbo Gao,Xuelong Li,Dacheng Tao

DOI: https://doi.org/10.1109/tsmcb.2012.2195171

2012-01-01

IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics)

Abstract:Conventional regression methods, such as multivariate linear regression (MLR) and its extension principal component regression (PCR), deal well with the situations that the data are of the form of low-dimensional vector. When the dimension grows higher, it leads to the under sample problem (USP): the dimensionality of the feature space is much higher than the number of training samples. However, little attention has been paid to such a problem. This paper first adopts an in-depth investigation to the USP in PCR, which answers three questions: 1) Why is USP produced? 2) What is the condition for USP, and 3) How is the influence of USP on regression. With the help of the above analysis, the principal components selection problem of PCR is presented. Subsequently, to address the problem of PCR, a multivariate multilinear regression (MMR) model is proposed which gives a substitutive solution to MLR, under the condition of multilinear objects. The basic idea of MMR is to transfer the multilinear structure of objects into the regression coefficients as a constraint. As a result, the regression problem is reduced to find two low-dimensional coefficients so that the principal components selection problem is avoided. Moreover, the sample size needed for solving MMR is greatly reduced so that USP is alleviated. As there is no closed-form solution for MMR, an alternative projection procedure is designed to obtain the regression matrices. For the sake of completeness, the analysis of computational cost and the proof of convergence are studied subsequently. Furthermore, MMR is applied to model the fitting procedure in the active appearance model (AAM). Experiments are conducted on both the carefully designed synthesizing data set and AAM fitting databases verified the theoretical analysis.
Comparison of variable selection methods in partial least squares regression

Tahir Mehmood,Solve Sæbø,Kristian Hovde Liland

DOI: https://doi.org/10.1002/cem.3226

IF: 2.5

2020-02-20

Journal of Chemometrics

Abstract:<p>Through the remarkable progress in technology, it is getting easier and easier to generate vast amounts of variables from a given sample. The selection of variables is imperative for data reduction and for understanding the modeled relationship. Partial least squares (PLS) regression is among the modeling approaches that address high throughput data. A considerable list of variable selection methods has been introduced in PLS. Most of these methods have been reviewed in a recently conducted study. Motivated by this, we have therefore conducted a comparison of available methods for variable selection within PLS. The main focus of this study was to reveal patterns of dependencies between variable selection method and data properties, which can guide the choice of method in practical data analysis. To this aim, a simulation study was conducted with data sets having diverse properties like the number of variables, the number of samples, model complexity level, and information content. The results indicate that the above factors like the number of variables, number of samples, model complexity level, information content and variant of PLS methods, and their mutual higher‐order interactions all significantly define the prediction capabilities of the model and the choice of variable selection strategy.</p>

chemistry, analytical,instruments & instrumentation,mathematics, interdisciplinary applications,automation & control systems,computer science, artificial intelligence,statistics & probability
Enhanced Model Predictions through Principal Components and Average Least Squares-Centered Penalized Regression

Adewale F. Lukman,Emmanuel T. Adewuyi,Ohud A. Alqasem,Mohammad Arashi,Kayode Ayinde

DOI: https://doi.org/10.3390/sym16040469

2024-04-13

Symmetry

Abstract:We address the estimation of regression parameters for the ill-conditioned predictive linear model in this study. Traditional least squares methods often encounter challenges in yielding reliable results when there is multicollinearity. Therefore, we employ a better shrinkage method, average least squares-centered penalized regression (ALPR), as it offers a more efficient approach for handling multicollinearity than ridge regression. Additionally, we integrate ALPR with the principal component (PC) dimension reduction method for enhanced performance. We compared the proposed PCALPR estimation technique with existing ones for ill-conditioned problems through comprehensive simulations and real-life data analyses using the mean squared error. This integration results in superior model performance compared to other methods, highlighting the potential of combining dimensionality reduction techniques with penalized regression for enhanced model predictions.

multidisciplinary sciences

Comparison of prediction methods for multicollinear data

A Two-Step PLS Inspired Method for Linear Prediction with Group Effect

Maximum Likelihood Estimators in a Two Step Model for PLS

Comparing methods addressing multi-collinearity when developing prediction models

Prediction of multivariate responses with a select number of principal components

Power Analysis of Principal Components Regression in Genetic Association Studies.

Performance of binary prediction models in high-correlation low-dimensional settings: a comparison of methods

Comparison of Canonical Correlation and Partial Least Squares analyses of simulated and empirical data

Partial Least Squares Methods: Partial Least Squares Correlation and Partial Least Square Regression

Improvement of simultaneous prediction using principal components approach

Principal component regression in NIR analysis: Viewpoints, background details and selection of components

Principal Model Analysis Based on Partial Least Squares

A correlation principal component regression analysis of NIR data

Partial least squares regression as an alternative to current regression methods used in ecology

Structured Principal Component Analysis Model With Variable Correlation Constraint

Principal Components Analysis in Clinical Studies

Comparison of variants of canonical correlation analysis and partial least squares for combined analysis of MRI and genetic data

Multivariate Multilinear Regression

Comparison of variable selection methods in partial least squares regression

Enhanced Model Predictions through Principal Components and Average Least Squares-Centered Penalized Regression