Abstract:Background: Residents receive a numeric performance rating (eg, 1-7 scoring scale) along with a narrative (ie, qualitative) feedback based on their performance in each workplace-based assessment (WBA). Aggregated qualitative data from WBA can be overwhelming to process and fairly adjudicate as part of a global decision about learner competence. Current approaches with qualitative data require a human rater to maintain attention and appropriately weigh various data inputs within the constraints of working memory before rendering a global judgment of performance. Objective: This study explores natural language processing (NLP) and machine learning (ML) applications for identifying trainees at risk using a large WBA narrative comment data set associated with numerical ratings. Methods: NLP was performed retrospectively on a complete data set of narrative comments (ie, text-based feedback to residents based on their performance on a task) derived from WBAs completed by faculty members from multiple hospitals associated with a single, large, residency program at McMaster University, Canada. Narrative comments were vectorized to quantitative ratings using the bag-of-n-grams technique with 3 input types: unigram, bigrams, and trigrams. Supervised ML models using linear regression were trained with the quantitative ratings, performed binary classification, and output a prediction of whether a resident fell into the category of at risk or not at risk. Sensitivity, specificity, and accuracy metrics are reported. Results: The database comprised 7199 unique direct observation assessments, containing both narrative comments and a rating between 3 and 7 in imbalanced distribution (scores 3-5: 726 ratings; and scores 6-7: 4871 ratings). A total of 141 unique raters from 5 different hospitals and 45 unique residents participated over the course of 5 academic years. When comparing the 3 different input types for diagnosing if a trainee would be rated low (ie, 1-5) or high (ie, 6 or 7), our accuracy for trigrams was 87%, bigrams 86%, and unigrams 82%. We also found that all 3 input types had better prediction accuracy when using a bimodal cut (eg, lower or higher) compared with predicting performance along the full 7-point rating scale (50%-52%). Conclusions: The ML models can accurately identify underperforming residents via narrative comments provided for WBAs. The words generated in WBAs can be a worthy data set to augment human decisions for educators tasked with processing large volumes of narrative assessments.

Scoring Dimension-Level Job Performance From Narrative Comments: Validity and Generalizability When Using Natural Language Processing

Harnessing Natural Language Processing to Support Decisions Around Workplace-Based Assessment: Machine Learning Study of Competency-Based Medical Education

Using natural language processing to increase prediction and reduce subgroup differences in personnel selection decisions.

Leveraging Narrative Feedback in Programmatic Assessment: The Potential of Automated Text Analysis to Support Coaching and Decision-Making in Programmatic Assessment

Automated scoring of the autobiographical interview with natural language processing

Using natural language processing to link patients' narratives to visual capabilities and sentiments

Machine-learning as a validated tool to characterize individual differences in free recall of naturalistic events

Assessing Students’ Use of Evidence and Organization in Response-to-Text Writing: Using Natural Language Processing for Rubric-Based Automated Scoring

Using Natural Language Processing to Visualize Narrative Feedback in a Medical Student Performance Dashboard

Natural Language Processing of Learners' Evaluations of Attendings to Identify Professionalism Lapses

Performance Analysis of a Serial Natural Language Processing Pipeline for Scaling Analytics of Academic Writing Process

Dimensional Modeling of Emotions in Text with Appraisal Theories: Corpus Creation, Annotation Reliability, and Prediction

Exploring a Language-Based Interest Assessment: Predicting Vocational Interests on Social Media Using Natural Language Processing

Measurement specificity with modern methods: Using dimensions, facets, and items from personality assessments to predict performance.

Question-based computational language approach outperforms rating scales in quantifying emotional states

An Equitable Framework for Automatically Assessing Children's Oral Narrative Language Abilities

LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores

Assessing the Performance of Human-Capable LLMs -- Are LLMs Coming for Your Job?

Gender Differences in Work-Based Assessment Scores and Narrative Comments After Direct Observation

Harnessing LLMs for multi-dimensional writing assessment: Reliability and alignment with human judgments

Graph-Based Word Alignment for Clinical Language Evaluation