Abstract:Background: A common feature of performance assessments is the use of human assessors to render judgements on student performance. From a measurement perspective, variability among assessors when assessing students may be viewed as a concern because it negatively impacts score reliability and validity. However, from a contextual perspective, variability among assessors is considered both meaningful and expected. A qualitative examination of assessor cognition when assessing student performance can assist in exploring what components are amenable to improvement through enhanced rater training, and the extent of variability when viewing assessors as contributing their individual expertise. Therefore, the purpose of this study was to explore assessor cognition as a source of score variability in a performance assessment of practice-based competencies. Method: A mixed-method sequential explanatory study design was used where findings from the qualitative strand assisted in the interpretation of results from the quantitative strand. Scores from one objective structured clinical examination (OSCE) were obtained for 95 occupational therapy students. Two Generalizability studies were conducted to examine the relative contribution of assessors as a source of score variability and to estimate the reliability of domain and holistic scores. Think-aloud interviews were conducted with eight participants assessing a subset of student performances from the OSCE in which they participated. Findings from the analysis of think-aloud data and consideration of assessors' background characteristics were used to assist in the interpretation of variance component estimates involving assessors, and score reliability. Results: Results from two generalizability analyses indicated the highest-order interaction-error term involving assessors accounted for the second-highest proportion of variance, after student variation. Score reliability was higher in the holistic vs. analytic scoring framework. Verbal analysis of assessors' think-aloud interviews provided evidential support for the quantitative results. Conclusions: This study provides insight into the nature and extent of assessor variability during a performance assessment of practice-based competencies. Study findings are interpretable from the measurement and contextual perspectives on assessor cognition. An integrated understanding is important to elucidate the meaning underlying the numerical score because the defensibility of inferences made about students' proficiencies rely on score quality, which in turn relies on expert judgements.

THE PSYCHOLOGY OF EDUCATIONAL MEASUREMENT

Dynamic measurement: A theoretical–psychometric paradigm for modern educational psychology

The Psychology of Academic Achievement

Measurement Models, Estimation, and the Study of Change

The measurement of psychological literacy: a first approximation

Assessing Efficiency in Education: The Imperative of Curriculum-Based Measurements

Refinement and Validation of Research Instrument for Assessing Executive Functioning Skills in the Post-Pandemic Education: The MEASURE Approach

Psychometric properties and factor analyses of the achievement motivation measure

Educational assessment without numbers

Shifting Conceptions of Validity in Educational Measurement: Implications for Performance Assessment

From Donders’ Dilemma to Objective Internal Assessment: How Experimental Developmental Psychology Can Contribute to Moral Education

Teaching Through Interactions in Secondary School Classrooms: Revisiting the Factor Structure and Practical Application of the Classroom Assessment Scoring System-Secondary

Measuring classroom management expertise (CME) of teachers: A video-based assessment approach and statistical results

Exploring cognitive behavioral aspects in educational psychology: A rigorous analysis of reliability and validity measures

Measurement Instruments When Comparing Academic Performance and Physical Activity: A Comprehensive Systematic Review

A Bottom-Up Approach to Measuring Student Social–Emotional Competence

Sociocognitive and Argumentation Perspectives on Psychometric Modeling in Educational Assessment

Improving Balance in Educational Measurement: A Legacy of E. F. Lindquist

Exploring assessor cognition as a source of score variability in a performance assessment of practice-based competencies

What happens when students reflect on their self-efficacy during a test? Exploring test experience and test outcome in science

On the measurement of competency