Abstract:Background: Although averaging across multiple examiners' judgements reduces unwanted overall score variability in objective structured clinical examinations (OSCE), designs involving several parallel circuits of the OSCE require that different examiner cohorts collectively judge performances to the same standard in order to avoid bias. Prior research suggests the potential for important examiner-cohort effects in distributed or national examinations that could compromise fairness or patient safety, but despite their importance, these effects are rarely investigated because fully nested assessment designs make them very difficult to study. We describe initial use of a new method to measure and adjust for examiner-cohort effects on students' scores. Methods: We developed video-based examiner score comparison and adjustment (VESCA): volunteer students were filmed 'live' on 10 out of 12 OSCE stations. Following the examination, examiners additionally scored station-specific common-comparator videos, producing partial crossing between examiner cohorts. Many-facet Rasch modelling and linear mixed modelling were used to estimate and adjust for examiner-cohort effects on students' scores. Results: After accounting for students' ability, examiner cohorts differed substantially in their stringency or leniency (maximal global score difference of 0.47 out of 7.0 [Cohen's d = 0.96]; maximal total percentage score difference of 5.7% [Cohen's d = 1.06] for the same student ability by different examiner cohorts). Corresponding adjustment of students' global and total percentage scores altered the theoretical classification of 6.0% of students for both measures (either pass to fail or fail to pass), whereas 8.6-9.5% students' scores were altered by at least 0.5 standard deviations of student ability. Conclusions: Despite typical reliability, the examiner cohort that students encountered had a potentially important influence on their score, emphasising the need for adequate sampling and examiner training. Development and validation of VESCA may offer a means to measure and adjust for potential systematic differences in scoring patterns that could exist between locations in distributed or national OSCE examinations, thereby ensuring equivalence and fairness.

Home vs Away Rotation Differences in the Standardized Letters of Evaluation (SLOE) 2.0

Comparing the Standardized Letter of Evaluation (SLOE) 2.0 with SLOE for Non-residency-based EM Physicians

Differences in Standardized Letter of Evaluation (SLOE) 2.0 Scoring Between Men and Women as well as Under-represented in Medicine and Non-underrepresented in Medicine Applicants

The Standardized Letter of Evaluation: How We Perceive the Quiet Student

The Emergency Medicine Group Standardized Letter of Evaluation as a Workplace-based Assessment: The Validity Is in the Detail

Decoding competitiveness: Exploring how emergency medicine faculty interpret standardized letters of evaluation

Faculty consensus on competitiveness for the new competency-based emergency medicine standardized letter of evaluation

Emergency Medicine Clerkship Grading Scheme, Grade, and Rank-List Distribution as Reported on Standardized Letters of Evaluation

EM Program Directors' Perception of "Lower 1/3rd" SLOEs in Ranking During the Matching Process

A Comparison of EFL Raters' Essay-Rating Processes across Two Types of Rating Scales.

The role of automated writing evaluation holistic scores in the ESL classroom

Digitizing Scoring Systems With Extended Online Feedback: A Novel Approach to Interactive Teaching and Learning in Formative OSCE

Perception of Quiet Students in Emergency Medicine: An Exploration of Narratives in the Standardized Letter of Evaluation

"Shadow" OSCE examiner. A cross-sectional study comparing the "shadow" examiner with the original OSCE examiner format

Using the Developmental Path of Cause to Bridge the Gap Between Awe Scores and Writing Teachers' Evaluations

Inter-school variations in the standard of examiners' graduation-level OSCE judgements

Developing a video-based method to compare and adjust examiner effects in fully nested OSCEs

Accountability in the evaluation of teacher effectiveness: views of teachers and administrators

Evaluation of Reading Level of Result Letters Sent to Patients from an Academic Primary Care Practice

Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking

Comparative Thematic Analysis of Emergency Medicine Standardized Letter of Evaluation Narrative Sections Between Chief Residents and Non-Chief Residents