Abstract:There has been a long term interest in using visual information to improve automatic speech recognition (ASR) system performance. Both audio and visual information are required in conventional audio visual speech recognition (AVSR) systems. This limits their wider applications when visual modality is not present. To this end, one possible solution is to use acoustic-to-visual (A2V) inversion techniques to generate visual features. Previous research in this direction used synthetic acoustic-articulatory parallel data in inversion model training. The acoustic mismatch between the audio-visual (AV) parallel data and target data was not considered. In addition, the target language to apply these technologies has been focused on English. In this article, a real 3D Audio-Visual Mandarin Continuous Speech (3DAV-MCS) corpus was used to train deep neural network based A2V inversion models. Cross-domain adaptation of the inversion models allows suitable visual features to be generated from acoustic data of mismatched domains. The proposed cross-domain deep visual feature generation techniques were evaluated on two state-of-the-art Mandarin speech recognition tasks: DAPRA GALE broadcast transcription and BOLT conversational telephone speech recognition. The AVSR systems constructed using the cross-domain generated visual features consistently outperformed the baseline convolutional neural network (CNN) ASR systems by up to 3.3% absolute (9.1% relative) character error rate (CER) reductions after both speaker adaptive training and sequence discriminative training were performed.

MAVD: The First Open Large-Scale Mandarin Audio-Visual Dataset with Depth Information

A Large-Scale Depth-Based Multimodal Audio-Visual Corpus in Mandarin

Deep Audio-visual System for Closed-set Word-level Speech Recognition

Audio-Visual Speech Recognition in MISP2021 Challenge: Dataset Release and Deep Analysis

AVA-AVD: Audio-Visual Speaker Diarization in the Wild

CN-CVS: A Mandarin Audio-Visual Dataset for Large Vocabulary Continuous Visual to Speech Synthesis

CI-AVSR: A Cantonese Audio-Visual Speech Datasetfor In-car Command Recognition

CI-AVSR: A Cantonese Audio-Visual Speech Dataset for In-car Command Recognition

SAVEn-Vid: Synergistic Audio-Visual Integration for Enhanced Understanding in Long Video Context

M$^3$AV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset

VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset

M3AV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset

MAViL: Masked Audio-Video Learners

M^3AV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset

MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation

MSDWild: Multi-modal Speaker Diarization Dataset in the Wild

Learning to Answer Questions in Dynamic Audio-Visual Scenarios

Cross-Domain Deep Visual Feature Generation for Mandarin Audio–Visual Speech Recognition

Audio-visual Speech Recognition Integrating 3D Lip Information Obtained from the Kinect

Robust Audio-visual Speech Recognition Using Bimodal Dfsmn with Multi-condition Training and Dropout Regularization.