Abstract:Background: A common feature of performance assessments is the use of human assessors to render judgements on student performance. From a measurement perspective, variability among assessors when assessing students may be viewed as a concern because it negatively impacts score reliability and validity. However, from a contextual perspective, variability among assessors is considered both meaningful and expected. A qualitative examination of assessor cognition when assessing student performance can assist in exploring what components are amenable to improvement through enhanced rater training, and the extent of variability when viewing assessors as contributing their individual expertise. Therefore, the purpose of this study was to explore assessor cognition as a source of score variability in a performance assessment of practice-based competencies. Method: A mixed-method sequential explanatory study design was used where findings from the qualitative strand assisted in the interpretation of results from the quantitative strand. Scores from one objective structured clinical examination (OSCE) were obtained for 95 occupational therapy students. Two Generalizability studies were conducted to examine the relative contribution of assessors as a source of score variability and to estimate the reliability of domain and holistic scores. Think-aloud interviews were conducted with eight participants assessing a subset of student performances from the OSCE in which they participated. Findings from the analysis of think-aloud data and consideration of assessors' background characteristics were used to assist in the interpretation of variance component estimates involving assessors, and score reliability. Results: Results from two generalizability analyses indicated the highest-order interaction-error term involving assessors accounted for the second-highest proportion of variance, after student variation. Score reliability was higher in the holistic vs. analytic scoring framework. Verbal analysis of assessors' think-aloud interviews provided evidential support for the quantitative results. Conclusions: This study provides insight into the nature and extent of assessor variability during a performance assessment of practice-based competencies. Study findings are interpretable from the measurement and contextual perspectives on assessor cognition. An integrated understanding is important to elucidate the meaning underlying the numerical score because the defensibility of inferences made about students' proficiencies rely on score quality, which in turn relies on expert judgements.

The stability of kindergarten teachers’ effectiveness: A generalizability study comparing the Framework For Teaching and the Classroom Assessment Scoring System

One size does not fit all: a concurrent analysis of the framework for teaching and the Classroom Strategies Assessment System

Measurement invariance of scores on the Teacher Stress Scale: international sample of PreK‐12 teachers

Assessing Teachers’ Classroom Management Competency: A Case Study of the Classroom Assessment Scoring System–Secondary

Psychometric Properties of the Classroom Assessment Scoring System (Pre-K): Implications for Measuring Interaction Quality in Diverse Early Childhood Settings

Exploring assessor cognition as a source of score variability in a performance assessment of practice-based competencies

Evaluating Special Education Instructional Practices Using Observation Rubrics: Investigating the Reliability of School Administrator Ratings

Don’t blame the teacher? The need to account for classroom characteristics in evaluations of teaching quality.

Assessing Quality of Teaching from Different Perspectives: Measurement Invariance across Teachers and Classes

The when and what of measuring ECE quality: Analysis of variation in the Classroom Assessment Scoring System (CLASS) across the ECE day

Inferring constructs of effective teaching from classroom observations: An application of Bayesian exploratory factor analysis without restrictions

Between-teacher variance of students' teacher-rated risk for emotional, behavioral, and adaptive functioning

The Mathematical Quality of Instruction (MQI) in Kindergarten: An Evaluation of the Stability of the MQI Using Generalizability Theory

Teaching Through Interactions in Secondary School Classrooms: Revisiting the Factor Structure and Practical Application of the Classroom Assessment Scoring System-Secondary

Assessing Noncognitive Aspects of School Readiness: The Predictive Validity of Brief Teacher Rating Scales of Social–Emotional Competence and Approaches to Learning

Examining Developmental Differences in Teachers' Observed Classroom Management Strategies Across Elementary, Middle, and High School

The Brief Student–Teacher Classroom Interaction Observation

Development and construct validity of the Classroom Strategies Scale-Observer Form.

Different Levels of Context-Specificity of Teacher Self-Efficacy and Their Relations With Teaching Quality

InClass – an instrument to assess classroom management in inclusive and special education with a focus on heterogeneous learning groups

Convergence Between Teacher Self-Report and School Administrator Observation Ratings Using the Classroom Strategies Assessment System