Abstract:Educational and Psychological Measurement, Ahead of Print. In educational assessment, cut scores are often defined through standard setting by a group of subject matter experts. This study aims to investigate the impact of several factors on classification accuracy using the receiver operating characteristic (ROC) analysis to provide statistical and theoretical evidence when the cut score needs to be refined. Factors examined in the study include the sample distribution relative to the cut score, prevalence of the positive event, and cost ratio. Forty item responses were simulated for examinees of four sample distributions. In addition, the prevalence and cost ratio between false negatives and false positives were manipulated to examine their impacts on classification accuracy. The optimal cut score is identified using the Youden Index J. The results showed that the optimal cut score identified by the evaluation criterion tended to pull the cut score closer to the mode of the proficiency distribution. In addition, depending on the prevalence of the positive event and cost ratio, the optimal cut score shifts accordingly. With the item parameters used to simulate the data and the simulated sample distributions, it was found that when passing the exam is a low-prevalence event in the population, increasing the cut score operationally improves the classification; when passing the exam is a high-prevalence event, then cut score should be reduced to achieve optimality. As the cost ratio increases, the optimal cut score suggested by the evaluation criterion decreases. In three out of the four sample distributions examined in this study, increasing the cut score enhanced the classification, irrespective of the cost ratio when the prevalence in the population is 50%. This study provides statistical evidence when the cut score needs to be refined for policy reasons.

Making model judgments ROC(K)-solid: Tailored cutoffs for fit indices through simulation and ROC analysis in structural equation modeling

dynamic : An R Package for Deriving Dynamic Fit Index Cutoffs for Factor Analysis

Dynamic fit index cutoffs for one-factor models

Dynamic fit index cutoffs for categorical factor analysis with Likert-type, ordinal, or binary responses.

Optimal Number of Replications for Obtaining Stable Dynamic Fit Index Cutoffs

Impact of methodological assumptions and covariates on the cutoff estimation in ROC analysis

Decision Curve Analysis: a Technical Note

Parsimony in Model Selection: Tools for Assessing Fit Propensity

A discrete time-to-event model for the meta-analysis of full ROC curves

Using ROC Analysis to Refine Cut Scores Following a Standard Setting Process

Evaluating Small Sample Approaches for Model Test Statistics in Structural Equation Modeling

Performance of Model Fit and Selection Indices for Bayesian Structural Equation Modeling with Missing Data

Decision Making with Machine Learning and ROC Curves

How robust are Structural Equation Models to model miss-specification? A simulation study

Fit Indices in Structural Equation Modeling and Confirmatory Factor Analysis: Reporting Guidelines

Robust and flexible inference for the covariate-specific ROC curve

Adapting fit indices for Bayesian structural equation modeling: Comparison to maximum likelihood.

The Accuracy of Bayesian Model Fit Indices in Selecting Among Multidimensional Item Response Theory Models

Cutoff criteria for overall model fit indexes in generalized structured component analysis

The Poor Fit of Model Fit for Selecting Number of Factors in Exploratory Factor Analysis for Scale Evaluation

Model-Selection-Based Approaches to Identifying the Optimal Number of Factors in Multilevel Exploratory Factor Analysis