A Computer-Assisted Tool for Automatically Measuring Non-Native Japanese Oral Proficiency
Wenchao Li,Zhentao Zhong,Haitao Liu
DOI: https://doi.org/10.1080/09588221.2024.2368838
IF: 7
2024-01-01
Computer Assisted Language Learning
Abstract:This study presents the development of an automated tool for assessing the speech of non-native Japanese speakers. The tool integrated an accent-trained automatic speech recording system based on deep neural networks, audio-text transcription, and natural language processing technologies to perform annotation, analysis, calculation, and scoring. A dataset consisting of audio recordings from 1000 Japanese learners across 16 countries was used for machine training. Validity and reliability tests were parallelly developed to evaluate: (a) the measures' ability to distinguish test takers with different oral proficiency levels; (b) the agreement between the annotations and scoring performed by the tool and human raters. The results, obtained through precision, recall, F-Score, quadratically-weighted kappa, proportional reduction of mean squared error, Pearson correlation, and multi-faceted Rasch measurement, demonstrated a relatively high reliability of the automated tool. The tool provides users with both a holistic score and individual scores for each criterion, allowing them to gain insights into their overall performance as well as detailed evaluations for each aspect of their proficiency. Consequently, it can be utilized for employment, education, and self-learning purposes.