Abstract:Endangered language generally has low-resource characteristics, as an immaterial cultural resource that cannot be renewed. Automatic speech recognition (ASR) is an effective means to protect this language. However, for low-resource language, native speakers are few and labeled corpora are insufficient. ASR, thus, suffers deficiencies including high speaker dependence and over fitting, which greatly harms the accuracy of recognition. To tackle the deficiencies, the paper puts forward an approach of audiovisual speech recognition (AVSR) based on LSTM-Transformer. The approach introduces visual modality information including lip movements to reduce the dependence of acoustic models on speakers and the quantity of data. Specifically, the new approach, through the fusion of audio and visual information, enhances the expression of speakers' feature space, thus achieving the speaker adaptation that is difficult in a single modality. The approach also includes experiments on speaker dependence and evaluates to what extent audiovisual fusion is dependent on speakers. Experimental results show that the CER of AVSR is 16.9% lower than those of traditional models (optimal performance scenario), and 11.8% lower than that for lip reading. The accuracy for recognizing phonemes, especially finals, improves substantially. For recognizing initials, the accuracy improves for affricates and fricatives where the lip movements are obvious and deteriorates for stops where the lip movements are not obvious. In AVSR, the generalization onto different speakers is also better than in a single modality and the CER can drop by as much as 17.2%. Therefore, AVSR is of great significance in studying the protection and preservation of endangered languages through AI.

Towards High Performance LVCSR in Speech-to-Speech Translation System on Smart Phones.

AudioVSR: Enhancing Video Speech Recognition with Audio Data

ViSPer: A Multilingual TTS Approach Based on VITS Using Deep Feature Loss

The IBM Speech-to-speech Translation System for Smartphone: Improvements for Resource-Constrained Tasks.

Phone Lattice Reconstruction for Embedded Language Recognition in LVCSR

Visual Information Assisted Mandarin Large Vocabulary Continuous Speech Recognition

Improving child speech recognition with augmented child-like speech

Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer

Cross-Lingual and Ensemble MLPs Strategies for Low-Resource Speech Recognition

Two-way speech-to-speech translation on handheld devices

Conformer-Based Speech Recognition On Extreme Edge-Computing Devices

Linguistic Search Optimization for Deep Learning Based LVCSR

Combination Of Data Borrowing Strategies For Low-Resource Lvcsr

Multi-Stream Posterior Features and Combining Subspace Gmms for Low Resource Lvcsr

DualVC 3: Leveraging Language Model Generated Pseudo Context for End-to-end Low Latency Streaming Voice Conversion

RefXVC: Cross-Lingual Voice Conversion with Enhanced Reference Leveraging

TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation

Improvement of Acoustic Models Fused with Lip Visual Information for Low-Resource Speech

R-BI: Regularized Batched Inputs enhance Incremental Decoding Framework for Low-Latency Simultaneous Speech Translation

Multilingual Speech Translation with Efficient Finetuning of Pretrained Models

Investigating Decoder-only Large Language Models for Speech-to-text Translation