Abstract:Introduction: The accurate assessment of health professionals' competence is critical for ensuring public health safety and quality of care. Computerized Adaptive Testing (CAT) based on the Item Response Theory (IRT) has the potential to improve measurement accuracy and reduce respondent burden. In this study, we conducted psychometric simulations to develop a CAT for evaluating the candidates' competence of health professionals. Methods: The initial CAT item bank was sourced from the Standardized Competence Test for Clinical Medicine Undergraduates (SCTCMU), a nationwide summative test in China, consisting of 300 multiple-choice items. We randomly selected response data from 2000 Chinese clinical medicine undergraduates for analysis. Two types of analyses were performed: first, evaluating the psychometric properties of all items to meet the requirements of CAT; and second, conducting multiple CAT simulations using both simulated and real response data. Results: The final CAT item bank consisted of 121 items, for which item parameters were calculated using a two-parameter logistic model (2PLM). The CAT simulations, based on both simulated and real data, revealed sufficient marginal reliability (coefficient of marginal reliability above 0.750) and criterion-related validity (Pearson's correlations between CAT scores and aggregate scores of the SCTCMU exceeding 0.850). Discussion: In national-level medical education assessment, there is an increasing need for concise yet valid evaluations of candidates' competence of health professionals. The CAT developed in this study demonstrated satisfactory reliability and validity, offering a more efficient assessment of candidates' competence of health professionals. The psychometric properties of the CAT could lead to shorter test durations, reduced information loss, and a decreased testing burden for participants.

A two‐step item bank calibration strategy based on 1‐bit matrix completion for small‐scale computerized adaptive testing

Online Item Calibration for Q-Matrix in CD-CAT.

Efficiency of computerized adaptive testing with a cognitively designed item bank

Methods for online calibration of Q-matrix and item parameters for polytomous responses in cognitive diagnostic computerized adaptive testing

A Robust Computerized Adaptive Testing Approach in Educational Question Retrieval

Novel item selection strategies for cognitive diagnostic computerized adaptive testing: A heuristic search framework

A Bounded Ability Estimation for Computerized Adaptive Testing

Design and Application of a Response-Driven Adaptive Test System

BanditCAT and AutoIRT: Machine Learning Approaches to Computerized Adaptive Testing and Item Calibration

Fully Adaptive Framework: Neural Computerized Adaptive Testing for Online Education.

Balancing Test Accuracy and Security in Computerized Adaptive Testing

A-Stratified Multistage Computerized Adaptive Testing

Ability Assessment Based on CAT in Adaptive Learning System

An adaptive testing item selection strategy via a deep reinforcement learning approach

Addressing Selection Bias in Computerized Adaptive Testing: A User-Wise Aggregate Influence Function Approach

Developing Computerized Adaptive Testing for a National Health Professionals Exam: An Attempt from Psychometric Simulations

Sequential Design for Computerized Adaptive Testing that Allows for Response Revision

Search-Efficient Computerized Adaptive Testing

Nonparametric CD‐CAT for multiple‐choice items: Item selection method and Q‐optimality

Applying Unidimensional and Multidimensional Item Response Theory Models in Testlet-Based Reading Assessment

Optimal Design for Item Calibration in Computerized Adaptive Testing: the 2PL Case