Abstract:End-to-end Automatic Speech Recognition (ASR) systems based on neural networks have seen large improvements in recent years. The availability of large scale hand-labeled datasets and sufficient computing resources made it possible to train powerful deep neural networks, reaching very low Word Error Rate (WER) on academic benchmarks. However, despite impressive performance on clean audio samples, a drop of performance is often observed on noisy speech. In this work, we propose to improve the noise robustness of the recently proposed Efficient Conformer Connectionist Temporal Classification (CTC)-based architecture by processing both audio and visual modalities. We improve previous lip reading methods using an Efficient Conformer back-end on top of a ResNet-18 visual front-end and by adding intermediate CTC losses between blocks. We condition intermediate block features on early predictions using Inter CTC residual modules to relax the conditional independence assumption of CTC-based models. We also replace the Efficient Conformer grouped attention by a more efficient and simpler attention mechanism that we call patch attention. We experiment with publicly available Lip Reading Sentences 2 (LRS2) and Lip Reading Sentences 3 (LRS3) datasets. Our experiments show that using audio and visual modalities allows to better recognize speech in the presence of environmental noise and significantly accelerate training, reaching lower WER with 4 times less training steps. Our Audio-Visual Efficient Conformer (AVEC) model achieves state-of-the-art performance, reaching WER of 2.3% and 1.8% on LRS2 and LRS3 test sets. Code and pretrained models are available at https://github.com/burchim/AVEC.

Fast Entropy-Based Methods of Word-Level Confidence Estimation for End-To-End Automatic Speech Recognition

BLSTM-Based Confidence Estimation for End-to-End Speech Recognition

Accurate and Reliable Confidence Estimation Based on Non-Autoregressive End-to-End Speech Recognition System

Fast Word Error Rate Estimation Using Self-Supervised Representations For Speech And Text

An Evolutionary Confidence Measure for Spotting Words in Speech Recognition

TeLeS: Temporal Lexeme Similarity Score to Estimate Confidence in End-to-End ASR

A Cross-Entropy-Guided (CEG) Measure for Speech Enhancement Front-End Assessing Performances of Back-End Automatic Speech Recognition

Audio-Visual Efficient Conformer for Robust Speech Recognition

Confidence-based Ensembles of End-to-End Speech Recognition Models

Confidence-features and Confidence-Scores for ASR Applications in Arbitration and DNN Speaker Adaptation.

Fast Context-Biasing for CTC and Transducer ASR models with CTC-based Word Spotter

Multi-Frame Cross-Entropy Training for Convolutional Neural Networks in Speech Recognition

Phonetic Clustering Based Confidence Measure for Embedded Speech Recognition

Self-consistent context aware conformer transducer for speech recognition

Improving Frame-level Classifier for Word Timings with Non-peaky CTC in End-to-End Automatic Speech Recognition

A maximum entropy approach to adaptive statistical language modelling

Empirically Combining Unnormalized NNLM and Back-off N -Gram for Fast N -Best Rescoring in Speech Recognition

A Novel Approach to Entropy-based Endpoint Detection of Noisy Speech

Lattice Based Transcription Loss for End-to-End Speech Recognition

Maximum Probability Increase Estimation Method for Fast Gaussian Likelihood Computations

A Novel Confidence Measure Based On Context Consistency For Spoken Term Detection