Abstract:Matrix sampling is a useful technique widely used in large-scale educational assessments. In an assessment with matrix sampling design, each examinee takes one of the multiple booklets with partial items. A critical problem of detecting differential item functioning (DIF) in such scenario has gained a lot of attention in recent years, which is, it is not appropriate to take the observed total score obtained from individual booklet as the matching variable in detecting the DIF. Therefore, the traditional detecting methods, such as Mantel-Haenszel (MH), SIBTEST, as well as Logistic Regression (LR) are not suitable. IRT_Δb might be an alternative due to its abilities to provide valid matching variable. However, the DIF classification criterion of IRT_Δb was not well established yet. Thus, the purpose of this study were: 1) to investigate the efficiency and robustness of using ability parameters obtained from Item Response Theory (IRT) model as the matching variable, comparing with the way using traditional observed raw total scores;2) to further identify what factors will influence the abilities in detecting DIF of two methods;3) to propose a DIF classification criteria for IRT_Δb. Simulated and empirical data were both employed in this study to explore the robustness and the efficiency of the two prevailing DIF detecting methods, which were the IRT_Δb method and the adapted LR method with the estimation of group-level ability based on IRT model as the matching variable. In the Monte Carlo study, a matrix sampling test was generated, and various experimental conditions were simulated as follows:1) different proportions of DIF items;2) different actual examinee ability distributions;3) different sample sizes;4) different size of DIF. Two DIF detection methods were then applied and results were compared. In addition, power functions were established in order to derive DIF classification rule for IRT_Δb based on current rules for LR. In the empirical study, through conducting a DIF analysis for American and Korean mathematics tests from Programme for International Student Assessment (PISA) 2003, the consistency of the classification rules between IRT_Δb and LR were further examined. The results indicated that in the matrix sampling design, both IRT_Δb method and adjusted LR method were sensitive to the diverse DIF magnitude. It was also found that the power, type I error, and the final classification of both methods were also influenced by the sample size, percentage of items with DIF, and ability differences between the focused group and the reference group. In conclusion, it was found that both the IRT_Δb method and adjusted LR method can be used to detect DIF in matrix sampling tests. A classification rule for IRT_Δb was proposed, which are: 0.85 between negligible DIF(A) and intermediate DIF(B), 1.23 between intermediate DIF(B) and large DIF(C). Meanwhile, it was suggested that researchers would take this rule as a tentative principle since the ΔR2 was limited between a narrow interval and the classification rule of LR was very flexible compared to classification rule of MH. Further studies could be conducted to take MH, IRT_Δb as well as LR into consideration simultaneously to give more comparable and consistent classification rules for different methods.

Standard Errors for National Trends in International Large-Scale Assessments in the Case of Cross-National Differential Item Functioning

Improving the Assessment of Differential Item Functioning in Large-Scale Programs With Dual-Scale Purification of Rasch Models: The PISA Example

Exploring Plausible Causes of Differential Item Functioning in the Pisa Science Assessment: Language, Curriculum or Culture

Testing Differential Item Functioning Without Predefined Anchor Items Using Robust Regression

Statistical Detection and Estimation of Differential Item Functioning in Computerized Adaptive Testing

Detecting Differential Item Functioning Using Response Time

Applying IRT_ΔB Procedure and Adapted LR Procedure to Detect DIF in Tests with Matrix Sampling

Exploring the Evidence to Interpret Differential Item Functioning via Response Process Data

Fairness and Comparability in Achievement Motivation Items: A Differential Item Functioning Analysis

Linking Large-Scale Reading Assessments: Measuring International Trends Over 40 Years

Correcting Selection Bias in Standardized Test Scores Comparisons

An Application of the Odds Ratios Method in Differential Item Functioning:An Illustration with an English Test

DIF Detection for Multiple Groups: Comparing Three‐Level GLMMs and Multiple‐Group IRT Models

Comparing Attitudes Across Groups: An IRT-Based Item-Fit Statistic for the Analysis of Measurement Invariance

Test Fairness: Examining Differential Functioning of the Reading Comprehension Section of the GSEEE in China

Using Interpretable Machine Learning for Differential Item Functioning Detection in Psychometric Tests

Measuring Motivation to Take Low-Stakes Large-Scale Test: New Model Based on Analyses of “Participant-Own-defined” Missingness

DIF Statistical Inference Without Knowing Anchoring Items

Differential Item Functioning via Robust Scaling

The use of test scores from large-scale assessment surveys: psychometric and statistical considerations

Modeling Item-Level Heterogeneous Treatment Effects With the Explanatory Item Response Model: Leveraging Large-Scale Online Assessments to Pinpoint the Impact of Educational Interventions