Abstract:Background: A common feature of performance assessments is the use of human assessors to render judgements on student performance. From a measurement perspective, variability among assessors when assessing students may be viewed as a concern because it negatively impacts score reliability and validity. However, from a contextual perspective, variability among assessors is considered both meaningful and expected. A qualitative examination of assessor cognition when assessing student performance can assist in exploring what components are amenable to improvement through enhanced rater training, and the extent of variability when viewing assessors as contributing their individual expertise. Therefore, the purpose of this study was to explore assessor cognition as a source of score variability in a performance assessment of practice-based competencies. Method: A mixed-method sequential explanatory study design was used where findings from the qualitative strand assisted in the interpretation of results from the quantitative strand. Scores from one objective structured clinical examination (OSCE) were obtained for 95 occupational therapy students. Two Generalizability studies were conducted to examine the relative contribution of assessors as a source of score variability and to estimate the reliability of domain and holistic scores. Think-aloud interviews were conducted with eight participants assessing a subset of student performances from the OSCE in which they participated. Findings from the analysis of think-aloud data and consideration of assessors' background characteristics were used to assist in the interpretation of variance component estimates involving assessors, and score reliability. Results: Results from two generalizability analyses indicated the highest-order interaction-error term involving assessors accounted for the second-highest proportion of variance, after student variation. Score reliability was higher in the holistic vs. analytic scoring framework. Verbal analysis of assessors' think-aloud interviews provided evidential support for the quantitative results. Conclusions: This study provides insight into the nature and extent of assessor variability during a performance assessment of practice-based competencies. Study findings are interpretable from the measurement and contextual perspectives on assessor cognition. An integrated understanding is important to elucidate the meaning underlying the numerical score because the defensibility of inferences made about students' proficiencies rely on score quality, which in turn relies on expert judgements.

STUDY OF SOURCES OF SCORE VARIABILITY IN PERFORMANCE ASSESSMENT USING MFRM : A CASE OF SPEAKING TESTIN PETS BAND3 ?

Investigating the reliability of CET-SET using Multi-Facet Rasch Model

Error Analysis of Inter-rater Reliability of the Second Language Writing Performance Assessment

The Measurement of Writing Scoring Quality with the Many-Facet Rasch Model

Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis

A Study on Assessing Errors in ESL Writing Scoring: A Multivariate Generalizability Theory Approach

A Many-Facet Rasch Measurement of Differential Rater Severity/Leniency in Three Types of Assessment

Application of Many-Facet Rasch Model in Rater Rraining for Subjective Items

Multi-faceted Rasch measurement and bias patterns in EFL writing performance assessment

Application of an Automated Essay Scoring engine to English writing assessment using Many-Facet Rasch Measurement

A Comparison of EFL Raters' Essay-Rating Processes across Two Types of Rating Scales.

Investigation of the effects of automatic scoring technology on human raters' performances in L2 speech proficiency assessment

Exploring assessor cognition as a source of score variability in a performance assessment of practice-based competencies

Multilevel Rasch Modeling of Two-Tier Multiple Choice Test: A Case Study Using Lawson’s Classroom Test of Scientific Reasoning

Comparing computer-based and paper-based rating modes in an English writing test

An IRT Analysis of Rater Bias in Structured Interview of National Civilian Candidates

Evaluating Differential Rater Functioning Over Time in the Context of Solo Music Performance Assessment

Investigating Rater Perceptions in the Assessment of Speaking

Investigating the Reliability of CET6 Essay Scoring:An Application of Generalizability Theory and Many-facet Rasch Model

AN INVESTIGATION OF SCORER RELIABILITY OF SELF AND PEER ASSESSMENT OF EFL WRITING AMONG CHINESE COLLEGE STUDENTS

An Empirical Study of the Inter - rater Reliability in Functional Movement Screen