Abstract:The general case of item preknowledge (IP) is studied, where groups of examinees had access to compromised subsets of items from an administered test prior to the exam. Nothing is known about these groups and subsets (terms groups and subsets are chosen to clearly distinguish between subsets of examinees and subsets of items). When only item scores are given, according to study by Karabatsos (Appl Meas Educ 16:277–298, 2003), the highest detection rate of examinees with IP is observed with TW statistic developed by Trabin and Weiss (in: Weiss (ed) New horizons in testing, Academic Press, New York, 1983). This paper develops a new Monte Carlo approach for detecting examinees with IP by estimating mean of a performance gain on a random sample of items (drawn from the administered test) relative to another random sample. Two samples are constructed such that for an examinee without IP, the gain should be low; meanwhile, for an examinee with IP, if the first sample has more compromised items than the second sample, the gain should not be low. Comparison study with TW using data simulating the general case of IP demonstrated a dramatic improvement of detection rates (over five times on average) when using the Monte Carlo approach. Even higher improvement (over 25 times) was observed in experiments with two publicly available real datasets. Recommendations for practitioners and extensions of the Monte Carlo approach are provided.

Using Item Scores and Distractors to Detect Item Compromise and Preknowledge

Detecting Item Preknowledge Using a Predictive Checking Method

Detection of Item Preknowledge Using Response Times

Are Exam Questions Known in Advance? Using Local Dependence to Detect Cheating

Detection of Item Preknowledge Using Likelihood Ratio Test and Score Test

Detecting Item Preknowledge Using Revisits With Speed and Accuracy

Monte Carlo detection of examinees with item preknowledge

A Machine-Learning-Based Approach for Detecting Item Preknowledge in Computerized Adaptive Testing

Detecting Compromised Items With Response Times Using a Bayesian Change-Point Approach

Sequential Detection of Compromised Items Using Response Times in Computerized Adaptive Testing

Comparing the Performance of Eight Item Preknowledge Detection Statistics

Which Statistic Should Be Used to Detect Item Preknowledge When the Set of Compromised Items Is Known?

Assessing Preknowledge Cheating via Innovative Measures: A Multiple-Group Analysis of Jointly Modeling Item Responses, Response Times, and Visual Fixation Counts

A Mixture Response Model for Identifying Item Preknowledge

A Mark-Recapture Approach to Estimating Item Pool Compromise

Detecting Preknowledge Cheating via Innovative Measures: A Mixture Hierarchical Model for Jointly Modeling Item Responses, Response Times, and Visual Fixation Counts

Two New Models for Item Preknowledge

Detecting Aberrant Behavior and Item Preknowledge: A Comparison of Mixture Modeling Method and Residual Method

The Impact of Item Preknowledge on Scaling and Equating: Item Response Theory True and Observed Score Equating Methods

Detecting Examinees With Item Preknowledge on Real Data

Using Deterministic, Gated Item Response Theory Model to detect test cheating due to item compromise