Abstract:End-to-end Automatic Speech Recognition (ASR) systems based on neural networks have seen large improvements in recent years. The availability of large scale hand-labeled datasets and sufficient computing resources made it possible to train powerful deep neural networks, reaching very low Word Error Rate (WER) on academic benchmarks. However, despite impressive performance on clean audio samples, a drop of performance is often observed on noisy speech. In this work, we propose to improve the noise robustness of the recently proposed Efficient Conformer Connectionist Temporal Classification (CTC)-based architecture by processing both audio and visual modalities. We improve previous lip reading methods using an Efficient Conformer back-end on top of a ResNet-18 visual front-end and by adding intermediate CTC losses between blocks. We condition intermediate block features on early predictions using Inter CTC residual modules to relax the conditional independence assumption of CTC-based models. We also replace the Efficient Conformer grouped attention by a more efficient and simpler attention mechanism that we call patch attention. We experiment with publicly available Lip Reading Sentences 2 (LRS2) and Lip Reading Sentences 3 (LRS3) datasets. Our experiments show that using audio and visual modalities allows to better recognize speech in the presence of environmental noise and significantly accelerate training, reaching lower WER with 4 times less training steps. Our Audio-Visual Efficient Conformer (AVEC) model achieves state-of-the-art performance, reaching WER of 2.3% and 1.8% on LRS2 and LRS3 test sets. Code and pretrained models are available at https://github.com/burchim/AVEC.

Bionic Cepstral coefficients (BCC): A new auditory feature extraction to noise-robust speaker identification

A novel hybrid feature method based on Caelen auditory model and gammatone filterbank for robust speaker recognition under noisy environment and speech coding distortion

Auditory model-based speech feature extraction and its application to speaker identification

Speech enhancement based on stationary bionic wavelet transform and maximum a posterior estimator of magnitude-squared spectrum

A Study of Acoustic Features in Arabic Speaker Identification under Noisy Environmental Conditions

Multimodal Speech Recognition Using EEG and Audio Signals: A Novel Approach for Enhancing ASR Systems

The application of bionic wavelet transform to speech signal processing in cochlear implants using neural network simulations

Bipolar Population Threshold Encoding for Audio Recognition with Deep Spiking Neural Networks

Auditory Model Based Speech Feature Extraction and Its Application to Speaker Identification

Robust Feature Extraction Using Temporal Context Averaging for Speaker Identification in Diverse Acoustic Environments

A Lightweight and Real-Time Binaural Speech Enhancement Model with Spatial Cues Preservation

Short Utterance Speaker Recognition Based on Speech High Frequency Information Compensation and Dynamic Feature Enhancement Methods

Comparison of feature extraction and normalization methods for speaker recognition using grid-audiovisual database

Wavelet-Based Mel-Frequency Cepstral Coefficients for Speaker Identification using Hidden Markov Models

Binaural cepstrum coefficient and its application to ground target recognition

An Auditory Feature Extraction Method Based on Forward-Masking and Its Application in Robust Speaker Identification and Speech Recognition.

Audio-Visual Efficient Conformer for Robust Speech Recognition

Improving Short-Duration Speaker Recognition by Joint Bark-Wavelet Acoustic Feature Coupling and Triplet Dual-Attention Mechanism Network

ELM speaker identification for limited dataset using multitaper based MFCC and PNCC features with fusion score

Modified MFCCs for Robust Speaker Recognition

Using Automatic Speech Recognition to Measure the Intelligibility of Speech Synthesized from Brain Signals