Automated Scoring of Open-Ended Situational Judgment Tests
Xu Jing,Luo Fang,Ma Yanzhen,Hu Luming,Tian Xuetao
DOI: https://doi.org/10.3724/sp.j.1041.2024.00831
2024-01-01
Acta Psychologica Sinica
Abstract:Situational Judgment Tests(SJTs)have gained popularity for their unique testing content and high face validity.However,traditional SJT formats,particularly those employing multiple-choice(MC)options,have encountered scrutiny due to their susceptibility to test-taking strategies.In contrast,open-ended and constructed response(CR)formats present a propitious means to address this issue.Nevertheless,their extensive adoption encounters hurdles primarily stemming from the financial implications associated with manual scoring.In response to this challenge,we propose an open-ended SJT employing a written-constructed response format for the assessment of teacher competency.This study established a scoring framework leveraging natural language processing(NLP)technology to automate the assessment of response texts,subsequently subjecting the system's validity to rigorous evaluation.The study constructed a comprehensive teacher competency model encompassing four distinct dimensions:student-oriented,problem-solving,emotional intelligence,and achievement motivation.Additionally,an open-ended situational judgment test was developed to gauge teachers'aptitude in addressing typical teaching dilemmas.A dataset comprising responses from 627 primary and secondary school teachers was collected,with manual scoring based on predefined criteria applied to 6,000 response texts from 300 participants.To expedite the scoring process,supervised learning strategies were employed,facilitating the categorization of responses at both the document and sentence levels.Various deep learning models,including the convolutional neural network(CNN),recurrent neural network(RNN),long short-term memory(LSTM),C-LSTM,RNN+attention,and LSTM+attention,were implemented and subsequently compared,thereby assessing the concordance between human and machine scoring.The validity of automatic scoring was also verified. This study reveals that the open-ended situational judgment test exhibited an impressive Cronbach's alpha coefficient of 0.91 and demonstrated a good fit in the validation factor analysis through the use of Mplus.Criterion-related validity was assessed,revealing significant correlations between test results and various educational facets,including instructional design,classroom evaluation,homework design,job satisfaction,and teaching philosophy.Among the diverse machine scoring models evaluated,CNNs have emerged as the top-performing model,boasting a scoring accuracy ranging from 70％to 88％,coupled with a remarkable degree of consistency with expert scores(r=0.95,QWK=0.82).The correlation coefficients between human and computer ratings for the four dimensions-student-oriented,problem-solving,emotional intelligence,and achievement motivation-approximated 0.9.Furthermore,the model showcased an elevated level of predictive accuracy when applied to new text datasets,serving as compelling evidence of its robust generalization capabilities. This study ventured into the realm of automated scoring for open-ended situational judgment tests,employing rigorous psychometric methodologies.To affirm its validity,the study concentrated on a specific facet:the evaluation of teacher competency traits.Fine-grained scoring guidelines were formulated,and state-of-the-art NLP techniques were used for text feature recognition and classification.The primary findings of this investigation can be summarized as follows:(1)Open-ended SJTs can establish precise scoring criteria grounded in crucial behavioral response elements;(2)Sentence-level text classification outperforms document-level classification,with CNNs exhibiting remarkable accuracy in response categorization;and(3)The scoring model consistently delivers robust performance and demonstrates a remarkable degree of alignment with human scoring,thereby hinting at its potential to partially supplant manual scoring procedures.