Abstract:Endangered language generally has low-resource characteristics, as an immaterial cultural resource that cannot be renewed. Automatic speech recognition (ASR) is an effective means to protect this language. However, for low-resource language, native speakers are few and labeled corpora are insufficient. ASR, thus, suffers deficiencies including high speaker dependence and over fitting, which greatly harms the accuracy of recognition. To tackle the deficiencies, the paper puts forward an approach of audiovisual speech recognition (AVSR) based on LSTM-Transformer. The approach introduces visual modality information including lip movements to reduce the dependence of acoustic models on speakers and the quantity of data. Specifically, the new approach, through the fusion of audio and visual information, enhances the expression of speakers' feature space, thus achieving the speaker adaptation that is difficult in a single modality. The approach also includes experiments on speaker dependence and evaluates to what extent audiovisual fusion is dependent on speakers. Experimental results show that the CER of AVSR is 16.9% lower than those of traditional models (optimal performance scenario), and 11.8% lower than that for lip reading. The accuracy for recognizing phonemes, especially finals, improves substantially. For recognizing initials, the accuracy improves for affricates and fricatives where the lip movements are obvious and deteriorates for stops where the lip movements are not obvious. In AVSR, the generalization onto different speakers is also better than in a single modality and the CER can drop by as much as 17.2%. Therefore, AVSR is of great significance in studying the protection and preservation of endangered languages through AI.

Map and Relabel: Towards Almost-Zero Resource Speech Recognition.

Maestro-U: Leveraging joint speech-text representation learning for zero supervised speech ASR

Multilingual Meta-Transfer Learning for Low-Resource Speech Recognition

Language-Universal Phonetic Representation in Multilingual Speech Pretraining for Low-Resource Speech Recognition

LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition

Deciphering Speech: a Zero-Resource Approach to Cross-Lingual Transfer in ASR

A reweighting method for speech recognition with imbalanced data of Mandarin and sub-dialects

M2ASR: Ambitions and First Year Progress

Consistency Based Unsupervised Self-training For ASR Personalisation

Unsupervised ASR via Cross-Lingual Pseudo-Labeling

Neural Model Reprogramming with Similarity Based Mapping for Low-Resource Spoken Command Recognition

Improved acoustic word embeddings for zero-resource languages using multilingual transfer

Low Resource Malay Dialect Automatic Speech Recognition Modeling Using Transfer Learning from a Standard Malay Model

Almost Unsupervised Text to Speech and Automatic Speech Recognition

From Semi-supervised to Almost-unsupervised Speech Recognition with Very-low Resource by Jointly Learning Phonetic Structures from Audio and Text Embeddings.

Almost-unsupervised Speech Recognition with Close-to-zero Resource Based on Phonetic Structures Learned from Very Small Unpaired Speech and Text Data.

Automatic Speech Recognition for Uyghur, Kazakh, and Kyrgyz: an Overview

Master-ASR: Achieving Multilingual Scalability and Low-Resource Adaptation in ASR with Modular Learning

Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels

Cross-Lingual and Ensemble MLPs Strategies for Low-Resource Speech Recognition

Improvement of Acoustic Models Fused with Lip Visual Information for Low-Resource Speech