Abstract:Several researchers have recommended that level-specific fit indices should be applied to detect the lack of model fit at any level in multilevel structural equation models. Although we concur with their view, we note that these studies did not sufficiently consider the impact of intraclass correlation (ICC) on the performance of level-specific fit indices. Our study proposed to fill this gap in the methodological literature. A Monte Carlo study was conducted to investigate the performance of (a) level-specific fit indices derived by a partially saturated model method (e.g., [Formula: see text] and [Formula: see text]) and (b) [Formula: see text] and [Formula: see text] in terms of their performance in multilevel structural equation models across varying ICCs. The design factors included intraclass correlation (ICC: ICC1 = 0.091 to ICC6 = 0.500), numbers of groups in between-level models (NG: 50, 100, 200, and 1,000), group size (GS: 30, 50, and 100), and type of misspecification (no misspecification, between-level misspecification, and within-level misspecification). Our simulation findings raise a concern regarding the performance of between-level-specific partial saturated fit indices in low ICC conditions: the performances of both [Formula: see text] and [Formula: see text] were more influenced by ICC compared with [Formula: see text] and SRMR B . However, when traditional cutoff values ( RMSEA≤ 0.06; CFI, TLI≥ 0.95; SRMR≤ 0.08) were applied, [Formula: see text] and [Formula: see text] were still able to detect misspecified between-level models even when ICC was as low as 0.091 (ICC1). On the other hand, both [Formula: see text] and [Formula: see text] were not recommended under low ICC conditions.

Value of sample size for computation of the Bayesian information criterion (BIC) in multilevel modeling

The Effective Sample Size in Bayesian Information Criterion for Level-Specific Fixed and Random Effects Selection in a Two-Level Nested Model

Choosing the number of factors in factor analysis with incomplete data via a novel hierarchical Bayesian information criterion

Model selection and psychological theory: A discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC).

Sufficient Sample Sizes for Multilevel Modeling

Model Selection for Mixtures of Factor Analyzers via Hierarchical BIC

A simple application of FIC to model selection

Bayesian variable selection using cost-adjusted BIC, with application to cost-effective measurement of quality of health care

Estimating Bayes factors from minimal summary statistics in repeated measures analysis of variance designs

Choosing the Number of Topics in LDA Models -- A Monte Carlo Comparison of Selection Criteria

Efficient Model Selection for Mixtures of Probabilistic PCA Via Hierarchical BIC.

Approaches for Quantifying the ICC in Multilevel Logistic Models: A Didactic Demonstration

Assessing an Alternative for `Negative Variance Components': A Gentle Introduction to Bayesian Covariance Structure Modelling for Negative Associations Among Patients with Personalized Treatments

The Effect of Small Sample Size on Two-Level Model Estimates: A Review and Illustration

The Impact of Intraclass Correlation on the Effectiveness of Level-Specific Fit Indices in Multilevel Structural Equation Modeling

Learning under Singularity: An Information Criterion improving WBIC and sBIC

A Bayesian Information Criterion for Portfolio Selection

Sufficient Sample Size and Power in Multilevel Ordinal Logistic Regression Models

Evaluation of Model Selection Strategies for Cross-Level Two-Way Differential Item Functioning Analysis

Review of sample size determination methods for the intraclass correlation coefficient in the one-way analysis of variance model

Statistical power in two-level models: A tutorial based on Monte Carlo simulation.