Abstract:Background: The ability of an expert's item difficulty ratings to predict test-taker actual performance is an important aspect of licensure examinations. Expert judgment is used as a primary source of information for users to make prior decisions to determine the pass rate of test takers. The nature of raters involved in predicting item difficulty is central to set credible standards. Therefore, this study aimed to assess and compare raters' prediction and actual Multiple-Choice Questions' difficulty of the undergraduate medicine licensure examination (UGMLE) in Ethiopia. Method: 815 examinees' responses to 200 Multiple-Choice Questions (MCQs) were used in this study. The study also included experts' item difficulty ratings of seven physicians who participated in the standard settings of UGMLE. Then, analysis was conducted to understand experts' rating variation in predicting the actual difficulty levels of examinees. Descriptive statistics was used to profile the mean rater's and actual difficulty value for MCQs, and ANOVA was used to compare the mean differences between raters' prediction of item difficulty. Additionally, regression analysis was used to understand the interrater variations in item difficulty predictions compared to the actual difficulty. The proportion of variance of actual difficulty explained from rater prediction was computed using regression analysis. Results: In this study, the mean difference between raters' prediction and examinees' actual performance was inconsistent across the exam domains. The study revealed a statistically significant strong positive correlation between the actual and predicted item difficulty in exam domains eight and eleven. However, a non-statistically significant very weak positive correlation was reported in exam domains seven and twelve. The multiple comparison analysis showed significant differences in mean item difficulty ratings between raters. In the regression analysis, experts' item difficulty ratings of the UGMLE had 33% power in predicting the actual difficulty level. The regression model also showed a moderate positive correlation (R = 0.57) that was statistically significant at F (6, 193) = 15.58, P = 0.001. Conclusion: This study demonstrated the complex process for assessing the difficulty level of MCQs in the UGMLE and emphasized the benefits of using experts' ratings in advance. To ensure the exams maintain the necessary reliable and valid scores, raters' accuracy on the UGMLE must be improved. To achieve this, techniques that align with the evolving assessment methodologies must be developed.

Item Analysis of Multiple Choice Questions in Medical Licensing Examination

Item analysis of multiple-choice questions in an undergraduate surgery course: An assessment of an assessment tool

Item Analysis of Multiple-choice Questions (MCQs): Assessment Tool For Quality Assurance Measures

Item Analysis of Single Best Response Type Multiple Choice Questions for Formative Assessment in Obstetrics and Gynaecology

Item Analysis of Multiple-Choice Question (MCQ)-Based Exam Efficiency Among Postgraduate Pediatric Medical Students: An Observational, Cross-Sectional Study From Saudi Arabia

Evaluation of medical assays by multiple-choice questions: Example of 10-years’ experience of a North African Country

Quality of multiple choice question items: item analysis

Item analysis of multiple-choice questions in pharmacology among medical undergraduates

Experts' prediction of item difficulty of multiple-choice questions in the Ethiopian Undergraduate Medicine Licensure Examination

Item and test analysis to identify quality multiple choice questions (MCQS) from an assessment of medical students of Ahmedabad, Gujarat

Item analysis of multiple choice questions of undergraduate pharmacology examinations in a medical college in Belagavi, Karnataka, India

Assessment of Global Health Education: The Role of Multiple-Choice Questions

Analysis of multiple-choice items

Analysis of Difficulty Index, Discrimination Index and Distractor Efficiency in Multiple Choice Questions

Associations between item characteristics and statistical performance for paediatric medical student multiple choice assessments

Quality Evaluation of Multiple-Choice Questions as an Assessment Tool for Medical Students

USE OF ITEM ANALYSIS OF MULTIPLE CHOICE QUESTIONS IN GENERAL PATHOLOGY

The Analysis of the Backgrounds and Scores of Examinees Entering for Clinical Licensing Examination Multi-Times

Postexamination item analysis of undergraduate pediatric multiple-choice questions exam: implications for developing a validated question Bank

Evaluating the Quality of Multiple-Choice Test Questions in the Postlicensure Environment

The legality and appropriateness of keeping Korean Medical Licensing Examination items confidential: a comparative analysis and review of court rulings