Abstract:End-to-end Automatic Speech Recognition (ASR) systems based on neural networks have seen large improvements in recent years. The availability of large scale hand-labeled datasets and sufficient computing resources made it possible to train powerful deep neural networks, reaching very low Word Error Rate (WER) on academic benchmarks. However, despite impressive performance on clean audio samples, a drop of performance is often observed on noisy speech. In this work, we propose to improve the noise robustness of the recently proposed Efficient Conformer Connectionist Temporal Classification (CTC)-based architecture by processing both audio and visual modalities. We improve previous lip reading methods using an Efficient Conformer back-end on top of a ResNet-18 visual front-end and by adding intermediate CTC losses between blocks. We condition intermediate block features on early predictions using Inter CTC residual modules to relax the conditional independence assumption of CTC-based models. We also replace the Efficient Conformer grouped attention by a more efficient and simpler attention mechanism that we call patch attention. We experiment with publicly available Lip Reading Sentences 2 (LRS2) and Lip Reading Sentences 3 (LRS3) datasets. Our experiments show that using audio and visual modalities allows to better recognize speech in the presence of environmental noise and significantly accelerate training, reaching lower WER with 4 times less training steps. Our Audio-Visual Efficient Conformer (AVEC) model achieves state-of-the-art performance, reaching WER of 2.3% and 1.8% on LRS2 and LRS3 test sets. Code and pretrained models are available at https://github.com/burchim/AVEC.

A Cross-Entropy-Guided (CEG) Measure for Speech Enhancement Front-End Assessing Performances of Back-End Automatic Speech Recognition

A Cross-Entropy-Guided Measure (CEGM) for Assessing Speech Recognition Performance and Optimizing DNN-Based Speech Enhancement

Acoustics-guided Evaluation (AGE): a New Measure for Estimating Performance of Speech Enhancement Algorithms for Robust ASR

Accurate and Reliable Confidence Estimation Based on Non-Autoregressive End-to-End Speech Recognition System

Single-Channel Speech Enhancement Algorithm Based on ME-MGCRN in Low Signal-to-Noise Scenario

SpeechColab Leaderboard: An Open-Source Platform for Automatic Speech Recognition Evaluation

Towards Decoupling Frontend Enhancement and Backend Recognition in Monaural Robust ASR

Audio-Visual Efficient Conformer for Robust Speech Recognition

Compensation of Speech Enhancement Distortion for Robust Speech Recognition

TeLeS: Temporal Lexeme Similarity Score to Estimate Confidence in End-to-End ASR

Comparing the Perceptual Contributions of Cochlear-Scaled Entropy and Speech Level

Time-Domain Speech Enhancement for Robust Automatic Speech Recognition

Improving the Intelligibility of Electric and Acoustic Stimulation Speech Using Fully Convolutional Networks Based Speech Enhancement

Correlated Multi-Level Speech Enhancement for Robust Real-World ASR Applications Using Mask-Waveform-Feature Optimization

Robust speaker recognition using glottal information‐based cepstral mean subtraction

A Weighted Cepstral Distance Measure for Objective Quality Evaluation of Mandarin

Improving the Intelligibility of Speech for Simulated Electric and Acoustic Stimulation Using Fully Convolutional Neural Networks

Multi-CMGAN+/+: Leveraging Multi-Objective Speech Quality Metric Prediction for Speech Enhancement

LPCSE: Neural Speech Enhancement through Linear Predictive Coding

Improving Speech Enhancement Performance by Leveraging Contextual Broad Phonetic Class Information

An improved hybrid CTC-Attention model for speech recognition