Abstract:Endangered language generally has low-resource characteristics, as an immaterial cultural resource that cannot be renewed. Automatic speech recognition (ASR) is an effective means to protect this language. However, for low-resource language, native speakers are few and labeled corpora are insufficient. ASR, thus, suffers deficiencies including high speaker dependence and over fitting, which greatly harms the accuracy of recognition. To tackle the deficiencies, the paper puts forward an approach of audiovisual speech recognition (AVSR) based on LSTM-Transformer. The approach introduces visual modality information including lip movements to reduce the dependence of acoustic models on speakers and the quantity of data. Specifically, the new approach, through the fusion of audio and visual information, enhances the expression of speakers' feature space, thus achieving the speaker adaptation that is difficult in a single modality. The approach also includes experiments on speaker dependence and evaluates to what extent audiovisual fusion is dependent on speakers. Experimental results show that the CER of AVSR is 16.9% lower than those of traditional models (optimal performance scenario), and 11.8% lower than that for lip reading. The accuracy for recognizing phonemes, especially finals, improves substantially. For recognizing initials, the accuracy improves for affricates and fricatives where the lip movements are obvious and deteriorates for stops where the lip movements are not obvious. In AVSR, the generalization onto different speakers is also better than in a single modality and the CER can drop by as much as 17.2%. Therefore, AVSR is of great significance in studying the protection and preservation of endangered languages through AI.

Transfer learning methods for low-resource speech accent recognition: A case study on Vietnamese language

Low Resource Malay Dialect Automatic Speech Recognition Modeling Using Transfer Learning from a Standard Malay Model

Unsupervised Pre-Training for Vietnamese Automatic Speech Recognition in the HYKIST Project

VAIS ASR: Building a conversational speech recognition system using language model combination

Multi-Dialect Vietnamese: Task, Dataset, Baseline Models and Challenges

Automatic Speech Recognition of Vietnamese for a New Large-Scale Corpus

Improvement of Acoustic Models Fused with Lip Visual Information for Low-Resource Speech

VietMed: A Dataset and Benchmark for Automatic Speech Recognition of Vietnamese in the Medical Domain

Efficient Utilization of Large Pre-Trained Models for Low Resource ASR

PhoWhisper: Automatic Speech Recognition for Vietnamese

Improving the transferability of speech separation by meta-learning

Low-Resource Language Identification From Speech Using Transfer Learning

Accent conversion using discrete units with parallel data synthesized from controllable accented TTS

Improving Vietnamese Named Entity Recognition from Speech Using Word Capitalization and Punctuation Recovery Models

Transfer Learning for Speech and Language Processing

Speech Technology for Everyone: Automatic Speech Recognition for Non-Native English with Transfer Learning

Real-Time End-to-End Speech Emotion Recognition with Cross-Domain Adaptation

ViDeBERTa: A powerful pre-trained language model for Vietnamese

Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models

ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing

Dialogue Act Segmentation for Vietnamese Human-Human Conversational Texts