Abstract:Introduction: The accurate assessment of health professionals' competence is critical for ensuring public health safety and quality of care. Computerized Adaptive Testing (CAT) based on the Item Response Theory (IRT) has the potential to improve measurement accuracy and reduce respondent burden. In this study, we conducted psychometric simulations to develop a CAT for evaluating the candidates' competence of health professionals. Methods: The initial CAT item bank was sourced from the Standardized Competence Test for Clinical Medicine Undergraduates (SCTCMU), a nationwide summative test in China, consisting of 300 multiple-choice items. We randomly selected response data from 2000 Chinese clinical medicine undergraduates for analysis. Two types of analyses were performed: first, evaluating the psychometric properties of all items to meet the requirements of CAT; and second, conducting multiple CAT simulations using both simulated and real response data. Results: The final CAT item bank consisted of 121 items, for which item parameters were calculated using a two-parameter logistic model (2PLM). The CAT simulations, based on both simulated and real data, revealed sufficient marginal reliability (coefficient of marginal reliability above 0.750) and criterion-related validity (Pearson's correlations between CAT scores and aggregate scores of the SCTCMU exceeding 0.850). Discussion: In national-level medical education assessment, there is an increasing need for concise yet valid evaluations of candidates' competence of health professionals. The CAT developed in this study demonstrated satisfactory reliability and validity, offering a more efficient assessment of candidates' competence of health professionals. The psychometric properties of the CAT could lead to shorter test durations, reduced information loss, and a decreased testing burden for participants.

Development and Validation of A Computer Adaptive Efl Test

Approach to Fitting Testlet for Computerized Adaptive Language Testing

Ability Assessment Based on CAT in Adaptive Learning System

Bridging Local Needs and National Standards: Use of Standards-Based Individualized Feedback of an In-House EFL Listening Test in China

A Robust Computerized Adaptive Testing Approach in Educational Question Retrieval

Test Fairness: Examining Differential Functioning of the Reading Comprehension Section of the GSEEE in China

Applying Unidimensional and Multidimensional Item Response Theory Models in Testlet-Based Reading Assessment

Fully Adaptive Framework: Neural Computerized Adaptive Testing for Online Education.

Computer Literacy and the Construct Validity of a High-Stakes Computer-Based Writing Assessment

Diagnosing listening and reading skills in the Chinese EFL context: Performance stability and variability across modalities and performance levels

Developing Computerized Adaptive Testing for a National Health Professionals Exam: An Attempt from Psychometric Simulations

Developing individualized feedback for listening assessment: Combining standard setting and cognitive diagnostic assessment approaches

Development and Validation of Computational Thinking Assessment of Chinese Elementary School Students

Validation Frameworks and Methods

A study of validity and reliability of Internet-based English writing testing

COMPUTER ADAPTIVE LANGUAGE TESTING ACCORDING TO NATO STANAG 6001 REQUIREMENTS

A Meaningful Attempt; Adopting Large Percentage of Subjective Items in College English Tests

A Preliminary Study of the Validity of the Internet-Based CET-4——Factors Affecting Test-takers' Perception of and Performance on the Test

The Assessment of Chinese Children’s English Vocabulary—A Culturally Appropriate Receptive Vocabulary Test for Young Chinese Learners of English

Diagnosing Primary Students’ Reading Progression: is Cognitive Diagnostic Computerized Adaptive Testing the Way Forward?

Efficiency of computerized adaptive testing with a cognitively designed item bank