Abstract:Endangered language generally has low-resource characteristics, as an immaterial cultural resource that cannot be renewed. Automatic speech recognition (ASR) is an effective means to protect this language. However, for low-resource language, native speakers are few and labeled corpora are insufficient. ASR, thus, suffers deficiencies including high speaker dependence and over fitting, which greatly harms the accuracy of recognition. To tackle the deficiencies, the paper puts forward an approach of audiovisual speech recognition (AVSR) based on LSTM-Transformer. The approach introduces visual modality information including lip movements to reduce the dependence of acoustic models on speakers and the quantity of data. Specifically, the new approach, through the fusion of audio and visual information, enhances the expression of speakers' feature space, thus achieving the speaker adaptation that is difficult in a single modality. The approach also includes experiments on speaker dependence and evaluates to what extent audiovisual fusion is dependent on speakers. Experimental results show that the CER of AVSR is 16.9% lower than those of traditional models (optimal performance scenario), and 11.8% lower than that for lip reading. The accuracy for recognizing phonemes, especially finals, improves substantially. For recognizing initials, the accuracy improves for affricates and fricatives where the lip movements are obvious and deteriorates for stops where the lip movements are not obvious. In AVSR, the generalization onto different speakers is also better than in a single modality and the CER can drop by as much as 17.2%. Therefore, AVSR is of great significance in studying the protection and preservation of endangered languages through AI.

ViSpeR: Multilingual Audio-Visual Speech Recognition

AudioVSR: Enhancing Video Speech Recognition with Audio Data

ViSPer: A Multilingual TTS Approach Based on VITS Using Deep Feature Loss

Cross-modal Mask Fusion and Modality-Balanced Audio-Visual Speech Recognition

XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception

VarASV: Enabling Pitch-variable Automatic Speaker Verification Via Multi-task Learning

Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer

Audio-Visual Speech Recognition in MISP2021 Challenge: Dataset Release and Deep Analysis

An arabic visual speech recognition framework with CNN and vision transformers for lipreading

Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whisper

Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition

Deep Audio-visual System for Closed-set Word-level Speech Recognition

Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels

Audio-Visual Speech Separation with Visual Features Enhanced by Adversarial Training

VILAS: Exploring the Effects of Vision and Language Context in Automatic Speech Recognition

Robust end-to-end deep audiovisual speech recognition

Large-vocabulary Audio-visual Speech Recognition in Noisy Environments

SAV-SE: Scene-aware Audio-Visual Speech Enhancement with Selective State Space Model

Improvement of Acoustic Models Fused with Lip Visual Information for Low-Resource Speech

SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Data

Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs