Abstract:The deficit in speech sound production in some children with autism spectrum disorder (ASD) adds to their communication barriers. The 3-D virtual environments have been implemented to improve their communication abilities. However, there were no previous studies on the use of a 3-D virtual pronunciation tutor designed specifically to train pronunciation for children with ASD. To fill this research gap, the current study developed and evaluated a 3-D virtual tutor which served as a multimodal and real-data-driven speech production tutor to present both places and manners of Mandarin articulation. Using an eye-tracking technique (RED 5 Eye Tracker), Experiment 1 objectively investigated children's gauged attention distribution online while learning with our computer-assisted 3-D virtual tutor in comparison to a real human face (HF) tutor. Eye-tracking results indicated most participants showed more interests in the visual speech cues of the 3-D tutor, and paid some degree of absolute attention to the additional visual speech information of both articulatory movements and airflow changes. To further compare treatment outcomes, training performance was evaluated in Experiment 2 with the ASD learners divided into two groups, with one group learning from the HF tutor and the other from the 3-D tutor (HF group vs. 3-D group). Both groups showed improvement with the help of computer-based training in the post-intervention test based on the calculation of a 5-point Likert scale. However, the 3-D group showed much higher gains in producing Mandarin stop and affricate consonants, and apical vowels. We conclude that our 3-D virtual imitation intervention system provides an effective approach of audiovisual pronunciation training for children with ASD.

Three-Dimensional Tongue Modeling Based on PCA for Visualized Pronunciation Feedback

Visualization of Mandarin articulation by using a physiological articulatory model

Real-time Ultrasound-enhanced Multimodal Imaging of Tongue using 3D Printable Stabilizer System: A Deep Learning Approach

PTeacher: a Computer-Aided Personalized Pronunciation Training System with Exaggerated Audio-Visual Corrective Feedback

An Improved 3D Geometric Tongue Model

Dynamic Extraction Method of 3D Parameters of Tongue for Pronunciation Recovery about Impaired Hearing Children

Analysis of and feedback on phonetic features in pronunciation training with a virtual teacher

A Speech-Driven 3-D Tongue Model with Realistic Movement in Mandarin Chinese.

Development and evaluation of a 3-D virtual pronunciation tutor for children with autism spectrum disorders

An English Pronunciation Evaluation Model Based on Acoustic Phonetics Features

Visualization of mandarin articulation driven by ultrasound data

Visual-speech Synthesis of Exaggerated Corrective Feedback

The Modeling of Tongue Tip in Standard Chinese Using MRI

A English Pronunciation Error Detection and Quality Evaluation Model Based on English Text

Combined X-ray and facial videos for phoneme-level articulator dynamics

Decoding the Dancing of the Tongue: A Model-Based Learning Approach to Phonetic Targets in Coarticulation

A Novel Method for Constructing 3d Geometric Articulatory Models

A Study On Robust Detection Of Pronunciation Erroneous Tendency Based On Deep Neural Network

3D dynamic tongue visualization based on DirectX

Effective Acoustic Modeling for Pronunciation Quality Scoring of Strongly Accented Mandarin Speech

Perceptual Evaluation of Pronunciation Quality for Computer Assisted Language Learning