Abstract:Theory: Assessment of clinical competence is a complex cognitive task with many mental demands often imposed on raters unintentionally. We were interested in whether this burden might contribute to well-described limitations in assessment judgments. In this study we examine the effect on indicators of rating quality of asking raters to (a) consider multiple competencies and (b) attend to multiple issues. In addition, we explored the cognitive strategies raters engage when asked to consider multiple competencies simultaneously. Hypotheses: We hypothesized that indications of rating quality (e.g., interrater reliability) would decline as the number of dimensions raters are expected to consider increases. Method: Experienced faculty examiners rated prerecorded clinical performances within a 2 (number of dimensions) × 2 (presence of distracting task) × 3 (number of videos) factorial design. Half of the participants were asked to rate 7 dimensions of performance (7D), and half were asked to rate only 2 (2D). The second factor involved the requirement (or lack thereof) to rate the performance of actors participating in the simulation. We calculated the interrater reliability of the scores assigned and counted the number of relevant behaviors participants identified as informing their ratings. Second, we analyzed data from semistructured posttask interviews to explore the rater strategies associated with rating under conditions designed to broaden raters' focus. Results: Generalizability analyses revealed that the 2D group achieved higher interrater reliability relative to the 7D group (G = .56 and .42, respectively, when the average of 10 raters is calculated). The requirement to complete an additional rating task did not have an effect. Using the 2 dimensions common to both groups, an analysis of variance revealed that participants who were asked to rate only 2 dimensions identified more behaviors of relevance to the focal dimensions than those asked to rate 7 dimensions: procedural skill = 36.2%, 95% confidence interval (CI) [32.5, 40.0] versus 23.5%, 95% CI [20.8, 26.3], respectively; history gathering = 38.6%, 95% CI [33.5, 42.9] versus 24.0%, 95% CI [21.1, 26.9], respectively; ps < .05. During posttask interviews, raters identified many sources of cognitive load and idiosyncratic cognitive strategies used to reduce cognitive load during the rating task. Conclusions: As intrinsic rating demands increase, indicators of rating quality decline. The strategies that raters engage when asked to rate many dimensions simultaneously are varied and appear to yield idiosyncratic efforts to reduce cognitive effort, which may affect the degree to which raters make judgments based on comparable information.

Beyond rating accuracy: unpacking frame-of-reference assessor training effectiveness

The Application of Frame-of-Reference Training on Nursing-Mini-CEX Rater Training

Faculty Perceptions of Frame of Reference Training to Improve Workplace-Based Assessment

Laypeople's beliefs about the effects of common estimator variables on memory

An Evaluation of Real-Time Feedback for Training Classroom Observers

Training as a performance appraisal improvement strategy

Investigation of 360-degree instrumentation effects: application of the Rasch rating scale model

Selecting and Simplifying: Rater Performance and Behavior When Considering Multiple Competencies

Exploring assessor cognition as a source of score variability in a performance assessment of practice-based competencies

An updated meta-analysis of the interrater reliability of supervisory performance ratings.

Perceptions of assessment center exercises: between exercises differences and interventions

How to Frame the Frame of Reference: A Comparison of Contextualization Methods

Self-Judgments of Accuracy as Indicators of Performance Evaluation Quality: Should We Believe Them?

A Comparison of EFL Raters' Essay-Rating Processes across Two Types of Rating Scales.

Reducing Attenuation Bias in Regression Analyses Involving Rating Scale Data via Psychometric Modeling

A comparison of the feedback sandwich, constructive-positive feedback, and within session feedback for training preference assessment implementation

Evaluating Differential Rater Functioning Over Time in the Context of Solo Music Performance Assessment

Investigating Rater Perceptions in the Assessment of Speaking

Transfer of Training: A Meta-Analytic Review

Advancing a Theory of Police Officer Training Motivation and Receptivity

Assessing supervisor versus trainee viewpoints of entrustment through cognitive and affective lenses: an artificial intelligence investigation of bias in feedback