Abstract:The ability to rapidly provide examinees with detailed and effective diagnostic information is a critical topic in psychology. Knowing what diagnostic criteria the examinees have met enables the practitioner to seek the solution to help them in a timely manner, and this can be achieved by cognitive diagnostic computerized adaptive testing (CD-CAT). However, the pervasive challenge of replenishing items in the CD-CAT item bank limits its practical application. Online calibration is a means to address item replenishment, but in CD-CAT, most existing online calibration methods that jointly calibrate the Q-matrix and item parameters of the new items are developed only for dichotomous responses and are time-consuming. Notably, previous studies pay no attention to polytomously scored items that are frequently observed in testing, even though they can offer additional evidence for the examinees' diagnosis. To fill this gap, we propose a SCAD-based method (SCAD-EM) to calibrate the Q-matrix and item parameters of the new items with polytomous response data in order to promote the application of CD-CAT in practice. The performance of the SCAD-EM was investigated in two comprehensive simulation studies and compared against the revised single-item estimation method (SIE-BIC). Results indicated that the SCAD-EM produces a higher calibration accuracy for the category-level Q-matrix and is computationally more efficient across all conditions, but it produces a lower calibration accuracy for the item-level Q-matrix. An empirical study further demonstrated the utility of the SCAD-EM and the SIE-BIC methods in calibrating new items with a real dataset. The advantages of the proposed method, its limitations, and possible future research directions are offered at the end.

Optimal Design for Item Calibration in Computerized Adaptive Testing: the 2PL Case

Nonlinear Sequential Designs for Logistic Item Response Theory Models with Applications to Computerized Adaptive Tests

Sequential Design for Computerized Adaptive Testing that Allows for Response Revision

Optimal Calibration of Items for Multidimensional Achievement Tests

Applying Unidimensional and Multidimensional Item Response Theory Models in Testlet-Based Reading Assessment

Approach to Fitting Testlet for Computerized Adaptive Language Testing

Parallel Optimal Calibration of Mixed-Format Items for Achievement Tests

Efficiency of computerized adaptive testing with a cognitively designed item bank

To Weight or Not to Weight? Balancing Influence of Initial Items in Adaptive Testing

Design and Application of a Response-Driven Adaptive Test System

A Robust Computerized Adaptive Testing Approach in Educational Question Retrieval

Optimal Designs for the Generalized Partial Credit Model

Nonparametric CD‐CAT for multiple‐choice items: Item selection method and Q‐optimality

Methods for online calibration of Q-matrix and item parameters for polytomous responses in cognitive diagnostic computerized adaptive testing

A two‐step item bank calibration strategy based on 1‐bit matrix completion for small‐scale computerized adaptive testing

Online Item Calibration for Q-Matrix in CD-CAT.

Exploration of Item Selection in Dual-Purpose Cognitive Diagnostic Computerized Adaptive Testing: Based on the RRUM.

A Comparative Study of On‐line Pretest Item—Calibration/Scaling Methods in Computerized Adaptive Testing

Improving the 3PLM-Based Computerized Adaptive Testing System with Multi-Agent Item Bank.

Modeling Rapid Guessing Behaviors in Computer-Based Testlet Items

Item response theory in high-stakes pharmacy assessments