Abstract:BackgroundThe goal of the Objective Structured Clinical Examination (OSCE) in Competency-based Medical Education (CBME) is to establish a minimal level of competence. The purpose of this study was to 1) to determine the credibility and acceptability of the modified Angoff method of standard setting in the setting of CBME, using the Borderline Group (BG) method and the Borderline Regression (BLR) method as a reference standard; 2) to determine if it is feasible to set different standards for junior and senior residents, and 3) to determine the desired characteristics of the judges applying the modified Angoff method.MethodsThe results of a previous OSCE study (21 junior residents, 18 senior residents, and six fellows) were used. Three groups of judges performed the modified Angoff method for both junior and senior residents: 1) sports medicine surgeons, 2) non-sports medicine orthopedic surgeons, and 3) sports fellows. Judges defined a borderline resident as a resident performing at a level between competent and a novice at each station. For each checklist item, the judges answered yes or no for “will the borderline/advanced beginner examinee respond correctly to this item?” The pass mark was calculated by averaging the scores. This pass mark was compared to that created using both the BG and the BLR methods.ResultsA paired t-test showed that all examiner groups expected senior residents to get significantly higher percentage of checklist items correct compared to junior residents (all stations p < 0.001). There were no significant differences due to judge type. For senior residents, there were no significant differences between the cut scores determined by the modified Angoff method and the BG/BLR method. For junior residents, the cut scores determined by the modified Angoff method were lower than the cut scores determined by the BG/BLR Method (all p < 0.01).ConclusionThe results of this study show that the modified Angoff method is an acceptable method of setting different pass marks for senior and junior residents. The use of this method enables both senior and junior residents to sit the same OSCE, preferable in the regular assessment environment of CBME.

Comparing Yes/No Angoff and Bookmark Standard Setting Methods in the Context of English Assessment

A Comparative Study of Pre-Service Teachers’ Views on Ethical Issues in Classroom Assessment in China and the United States

A Comparison of EFL Raters' Essay-Rating Processes across Two Types of Rating Scales.

Bridging Local Needs and National Standards: Use of Standards-Based Individualized Feedback of an In-House EFL Listening Test in China

A Contrastive Analysis of Classroom-Based Language Assessments

Review on the validity of China’s Standards of English Language Ability

Setting standards for teaching evaluation data: an application of the contrasting groups method

Comparing computer-based and paper-based rating modes in an English writing test

Development and Validation of A Computer Adaptive Efl Test

How to set the bar in competency-based medical education: standard setting after an Objective Structured Clinical Examination (OSCE)

Establishing Comparability Across Writing Tasks With Picture Prompts of Three Alternate Tests

A Study of English Teaching Differences Between China and The United States

Examining mode effects for an adapted Chinese critical thinking assessment

An Empirical Evaluation of Two College English Textbooks in China Based on the China Standards English Framework

Assessing the Comparability of School Scores Across Test Forms That Are Not Parallel

Using ROC Analysis to Refine Cut Scores Following a Standard Setting Process

One Score to Rule Them All? Comparing the Predictive and Concurrent Validity of 30 Hearts and Flowers Scoring Approaches

Standard-Setting Methods for Assessment in a Post-Graduate Medical College

A CASE STUDY OF FORMATIVE ASSESSMENT IN A CHINESE HIGH SCHOOL

EXAMINING THE CONTENT ALIGNMENT BETWEEN LANGUAGE CURRICULUM AND A LANGUAGE TEST IN CHINA

Comparative approaches to the assessment of writing: Reliability and validity of benchmark rating and comparative judgement