Abstract:Background: A common feature of performance assessments is the use of human assessors to render judgements on student performance. From a measurement perspective, variability among assessors when assessing students may be viewed as a concern because it negatively impacts score reliability and validity. However, from a contextual perspective, variability among assessors is considered both meaningful and expected. A qualitative examination of assessor cognition when assessing student performance can assist in exploring what components are amenable to improvement through enhanced rater training, and the extent of variability when viewing assessors as contributing their individual expertise. Therefore, the purpose of this study was to explore assessor cognition as a source of score variability in a performance assessment of practice-based competencies. Method: A mixed-method sequential explanatory study design was used where findings from the qualitative strand assisted in the interpretation of results from the quantitative strand. Scores from one objective structured clinical examination (OSCE) were obtained for 95 occupational therapy students. Two Generalizability studies were conducted to examine the relative contribution of assessors as a source of score variability and to estimate the reliability of domain and holistic scores. Think-aloud interviews were conducted with eight participants assessing a subset of student performances from the OSCE in which they participated. Findings from the analysis of think-aloud data and consideration of assessors' background characteristics were used to assist in the interpretation of variance component estimates involving assessors, and score reliability. Results: Results from two generalizability analyses indicated the highest-order interaction-error term involving assessors accounted for the second-highest proportion of variance, after student variation. Score reliability was higher in the holistic vs. analytic scoring framework. Verbal analysis of assessors' think-aloud interviews provided evidential support for the quantitative results. Conclusions: This study provides insight into the nature and extent of assessor variability during a performance assessment of practice-based competencies. Study findings are interpretable from the measurement and contextual perspectives on assessor cognition. An integrated understanding is important to elucidate the meaning underlying the numerical score because the defensibility of inferences made about students' proficiencies rely on score quality, which in turn relies on expert judgements.

Overview of couplet scoring in content-focused physics assessments

Coupled Multiple-response vs. Free-response Conceptual assessment: An Example from upper-division Physics

Rubric Design for Separating the Roles of Open-Ended Assessments

Validation and analysis of the coupled multiple response Colorado upper-division electrostatics (CUE) diagnostic

Development and Uses of Upper-division Conceptual Assessment

Assessing Student Learning in Middle-Division Classical Mechanics/Math Methods

Assessing Learning Outcomes in Middle-Division Classical Mechanics: The Colorado Classical Mechanics/Math Methods Instrument

What are we assessing? An analysis of the most common concept inventories in physics

Evaluating Different Scoring Methods for Multiple Response Items Providing Partial Credit

A comparison of integrated testlet and constructed-response question formats

A Comprehensive Assessment Strategy for Physics Laboratory Courses

Assessing the quality of a student-generated question repository

Student-Created Physics Problems as an Independent and Equitable Assessment Tool

Comparing student performance on a multi-attempt asynchronous assessment to a single-attempt synchronous assessment in introductory level physics

A Polytomous Scoring Approach to Handle Not-Reached Items in Low-Stakes Assessments

Using Automated Indices of Cohesion to Evaluate an Intelligent Tutoring System and an Automated Writing Evaluation System

Grading by Category: A simple method for providing students with meaningful feedback on exams in large courses

Assessing Scientific Practices in Physics Paper-based Assessments

Adaptive testing with the GGUM-RANK multidimensional forced choice model: Comparison of pair, triplet, and tetrad scoring

Exploring assessor cognition as a source of score variability in a performance assessment of practice-based competencies

Composition Organization and Development Analysis (CODA) Scale: Equipping high school students to evaluate argumentative essays