Abstract:Partial least squares path modeling (PLS-PM) has become popular in various disciplines to model structural relationships among latent variables measured by manifest variables. To fully benefit from the predictive capabilities of PLS-PM, researchers must understand the efficacy of predictive metrics used. In this research, we compare the performance of standard PLS-PM criteria and model selection criteria derived from Information Theory, in terms of selecting the best predictive model among a cohort of competing models. We use Monte Carlo simulation to study this question under various sample sizes, effect sizes, item loadings, and model setups. Specifically, we explore whether, and when, the in-sample measures such as the model selection criteria can substitute for out-of-sample criteria that require a holdout sample. Such a substitution is advantageous when creating a holdout causes considerable loss of statistical and predictive power due to an overall small sample. We find that when the researcher does not have the luxury of a holdout sample, and the goal is selecting correctly specified models with low prediction error, the in-sample model selection criteria, in particular the Bayesian Information Criterion (BIC) and Geweke-Meese Criterion (GM), are useful substitutes for out-of-sample criteria. When a holdout sample is available, the best performing out-of-sample criteria include the root mean squared error (RMSE) and mean absolute deviation (MAD). We recommend against using standard the PLS-PM criteria (R-2, Adjusted R-2, and Q(2)), and specifically the out-of-sample mean absolute percentage error (MAPE) for prediction-oriented model selection purposes. Finally, we illustrate the model selection criteria's practical utility using a well-known corporate reputation model.

LOO and WAIC as Model Selection Methods for Polytomous Items

Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC

Variable Selection with Stepwise and Best Subset Approaches.

An Information Analysis on Modeling Interaction Effects in Logistic Regression

The Accuracy of Bayesian Model Fit Indices in Selecting Among Multidimensional Item Response Theory Models

Empirical-likelihood-based criteria for model selection on marginal analysis of longitudinal data with dropout missingness

Evaluation of Model Selection Strategies for Cross-Level Two-Way Differential Item Functioning Analysis

The Best Fit Bayesian Hierarchical Generalized Linear Model Selection Using Information Complexity Criteria in the MCMC Approach

Extending AIC to best subset regression

Are information criteria good enough to choose the right the number of regimes in hidden Markov models?

Prediction-Oriented Model Selection In Partial Least Squares Path Modeling

A Comparative Study of Imputation Methods for Multivariate Ordinal Data

Decompositions of Information Criteria in Joint Model Under Probit Link for Response

Consistent Model Selection Procedure for Random Coefficient INAR Models

A Comparison of Limited Information Estimation Methods for the Two-Parameter Normal-Ogive Model with Locally Dependent Items

Confirmatory Factor Analysis of Ordinal Variables With Misspecified Models

PanIC: consistent information criteria for general model selection problems

Model Selection for Mixtures of Factor Analyzers via Hierarchical BIC

Making model judgments ROC(K)-solid: Tailored cutoffs for fit indices through simulation and ROC analysis in structural equation modeling

Interchangeability between factor analysis, logistic IRT, and normal ogive IRT