Abstract:Multilevel modeling (MLM) is commonly used in psychological research to model clustered data. However, data in applied research usually violate one of the essential assumptions of MLM—homogeneity of variance. While the fixed-effect estimates produced by the maximum likelihood method remain unbiased, the standard errors for the fixed effects are misestimated, resulting in inaccurate inferences and inflated or deflated type I error rates. To correct the bias in fixed effects standard errors and provide valid inferences, small-sample corrections such as the Kenward-Roger (KR) adjustment and the adjusted cluster-robust standard errors (CR-SEs) with the Satterthwaite approximation for t tests have been used. The current study compares KR with random slope (RS) models and the adjusted CR-SEs with ordinary least squares (OLS), random intercept (RI) and RS models to analyze small, heteroscedastic, clustered data using a Monte Carlo simulation. Results show the KR procedure with RS models has large biases and inflated type I error rates for between-cluster effects in the presence of level 2 heteroscedasticity. In contrast, the adjusted CR-SEs generally yield results with acceptable biases and maintain type I error rates close to the nominal level for all examined models. Thus, when the interest is only in within-cluster effect, any model with the adjusted CR-SEs could be used. However, when the interest is to make accurate inferences of the between-cluster effect, researchers should use the adjusted CR-SEs with RS to have higher power and guard against unmodeled heterogeneity. We reanalyzed an example in Snijders & Bosker (2012) to demonstrate the use of the adjusted CR-SEs with different models.

Residual Normality Assumption and the Estimation of Multiple Membership Random Effects Models

Using Multilevel Models and Generalized Estimating Equation Models to Account for Clustering in Neurology Clinical Research

Inferential Tools for Assessing Dependence Across Response Categories in Multinomial Models with Discrete Random Effects

Mixed‐Effect Hybrid Models for Longitudinal Data with Nonignorable Dropout

Evaluating two small-sample corrections for fixed-effects standard errors and inferences in multilevel models with heteroscedastic, unbalanced, clustered data

Estimation for an improved multilevel model based on MCMC algorithm

Multilevel multinomial logit regression model with random effects: application to flash EuroBarometer euro survey data

Statistical Inference in Generalized Linear Mixed Models by Joint Modelling Mean and Covariance of Non-Normal Random Effects

Semiparametric Quantile Modelling of Hierarchical Data

Maximum-Likelihood Estimation and Influence Analysis in Multivariate Skew-Normal Reproductive Dispersion Mixed Models for Longitudinal Data

Multiple membership multilevel models

A Monte Carlo Study of the Effects of Common Method Variance on Significance Testing and Parameter Bias in Hierarchical Linear Modeling

A demonstration and evaluation of the use of cross-classified random-effects models for meta-analysis

The Effect of Latent and Error Non-Normality on Measures of Fit in Structural Equation Modeling

Estimation for Improved Multilevel Model via the EM Algorithm

Checking for Normality in Linear Mixed Models

Improved confidence intervals for nonlinear mixed-effects and nonparametric regression models

Bayesian Inference for Skew-Normal Nonlinear Reproductive Dispersion Mixed-Effect Models

Linear Mixed-Effects Models for Dependent Data: Power and Accuracy in Parameter Estimation.

On the impact of parametric assumptions and robust alternatives for longitudinal data analysis.

A Varying-Coefficient Approach to Estimating Multi-Level Clustered Data Models