Abstract:In educational and psychological measurement, researchers and/or practitioners are often interested in examining whether the ability of an examinee is the same over two sets of items. Such problems can arise in measurement of change, detection of cheating on unproctored tests, erasure analysis, detection of item preknowledge, etc. Traditional frequentist approaches that are used in such problems include the Wald test, the likelihood ratio test, and the score test (e.g., Fischer, Appl Psychol Meas 27:3-26, 2003; Finkelman, Weiss, & Kim-Kang, Appl Psychol Meas 34:238-254, 2010; Glas & Dagohoy, Psychometrika 72:159-180, 2007; Guo & Drasgow, Int J Sel Assess 18:351-364, 2010; Klauer & Rettig, Br J Math Stat Psychol 43:193-206, 1990; Sinharay, J Educ Behav Stat 42:46-68, 2017). This paper shows that approaches based on higher-order asymptotics (e.g., Barndorff-Nielsen & Cox, Inference and asymptotics. Springer, London, 1994; Ghosh, Higher order asymptotics. Institute of Mathematical Statistics, Hayward, 1994) can also be used to test for the equality of the examinee ability over two sets of items. The modified signed likelihood ratio test (e.g., Barndorff-Nielsen, Biometrika 73:307-322, 1986) and the Lugannani-Rice approximation (Lugannani & Rice, Adv Appl Prob 12:475-490, 1980), both of which are based on higher-order asymptotics, are shown to provide some improvement over the traditional frequentist approaches in three simulations. Two real data examples are also provided.

Detection of Item Preknowledge Using Likelihood Ratio Test and Score Test

Detection of Item Preknowledge Using Response Times

Which Statistic Should Be Used to Detect Item Preknowledge When the Set of Compromised Items Is Known?

Detecting Item Preknowledge Using a Predictive Checking Method

A Comparison of Methods for Detecting Examinee Preknowledge of Items

Detecting Item Preknowledge Using Revisits With Speed and Accuracy

Using Item Scores and Distractors to Detect Item Compromise and Preknowledge

Comparing the Performance of Eight Item Preknowledge Detection Statistics

A Machine-Learning-Based Approach for Detecting Item Preknowledge in Computerized Adaptive Testing

A Mixture Response Model for Identifying Item Preknowledge

Detecting Preknowledge Cheating via Innovative Measures: A Mixture Hierarchical Model for Jointly Modeling Item Responses, Response Times, and Visual Fixation Counts

Detecting Aberrant Behavior and Item Preknowledge: A Comparison of Mixture Modeling Method and Residual Method

Two New Models for Item Preknowledge

Monte Carlo detection of examinees with item preknowledge

Assessing Preknowledge Cheating via Innovative Measures: A Multiple-Group Analysis of Jointly Modeling Item Responses, Response Times, and Visual Fixation Counts

Concurrent Use of Response Time and Response Accuracy for Detecting Examinees with Item Preknowledge

Robustness of Computer Adaptive Tests to the Presence of Item Preknowledge: A Simulation Study

Higher-Order Asymptotics and Its Application to Testing the Equality of the Examinee Ability Over Two Sets of Items

Are Exam Questions Known in Advance? Using Local Dependence to Detect Cheating

On the Equivalence of a Likelihood Ratio of Drasgow, Levine, and Zickar (1996) and the Statistic Based on the Neyman-Pearson Lemma of Belov (2016)

The Lack of Robustness of a Statistic Based on the Neyman-Pearson Lemma to Violations of Its Underlying Assumptions