Abstract:Endangered language generally has low-resource characteristics, as an immaterial cultural resource that cannot be renewed. Automatic speech recognition (ASR) is an effective means to protect this language. However, for low-resource language, native speakers are few and labeled corpora are insufficient. ASR, thus, suffers deficiencies including high speaker dependence and over fitting, which greatly harms the accuracy of recognition. To tackle the deficiencies, the paper puts forward an approach of audiovisual speech recognition (AVSR) based on LSTM-Transformer. The approach introduces visual modality information including lip movements to reduce the dependence of acoustic models on speakers and the quantity of data. Specifically, the new approach, through the fusion of audio and visual information, enhances the expression of speakers' feature space, thus achieving the speaker adaptation that is difficult in a single modality. The approach also includes experiments on speaker dependence and evaluates to what extent audiovisual fusion is dependent on speakers. Experimental results show that the CER of AVSR is 16.9% lower than those of traditional models (optimal performance scenario), and 11.8% lower than that for lip reading. The accuracy for recognizing phonemes, especially finals, improves substantially. For recognizing initials, the accuracy improves for affricates and fricatives where the lip movements are obvious and deteriorates for stops where the lip movements are not obvious. In AVSR, the generalization onto different speakers is also better than in a single modality and the CER can drop by as much as 17.2%. Therefore, AVSR is of great significance in studying the protection and preservation of endangered languages through AI.

Lrspeech: Extremely low-resource speech synthesis and recognition

LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition

End-to-end Text-to-speech for Low-resource Languages by Cross-Lingual Transfer Learning.

Enhancing Low-Resource ASR through Versatile TTS: Bridging the Data Gap

Almost Unsupervised Text to Speech and Automatic Speech Recognition

Zero-resource Speech Translation and Recognition with LLMs

Effective Training End-to-End ASR systems for Low-resource Lhasa Dialect of Tibetan Language

Pre-training Techniques for Improving Text-to-Speech Synthesis by Automatic Speech Recognition Based Data Enhancement

Low-Resource Mongolian Speech Synthesis Based on Automatic Prosody Annotation

Towards High-Quality Neural TTS for Low-Resource Languages by Learning Compact Speech Representations

Text-to-Speech for Low-Resource Agglutinative Language With Morphology-Aware Language Model Pre-Training

Optimizing Data Usage for Low-Resource Speech Recognition

A multilingual training strategy for low resource Text to Speech

Creating Spoken Dialog Systems in Ultra-Low Resourced Settings

ELAICHI: Enhancing Low-resource TTS by Addressing Infrequent and Low-frequency Character Bigrams

SR-TTS: a rhyme-based end-to-end speech synthesis system

Exploring Effective Data Utilization for Low-Resource Speech Recognition

Map and Relabel: Towards Almost-Zero Resource Speech Recognition.

Rapid Speaker Adaptation in Low Resource Text to Speech Systems using Synthetic Data and Transfer learning

Improvement of Acoustic Models Fused with Lip Visual Information for Low-Resource Speech