Abstract:End-to-end Automatic Speech Recognition (ASR) systems based on neural networks have seen large improvements in recent years. The availability of large scale hand-labeled datasets and sufficient computing resources made it possible to train powerful deep neural networks, reaching very low Word Error Rate (WER) on academic benchmarks. However, despite impressive performance on clean audio samples, a drop of performance is often observed on noisy speech. In this work, we propose to improve the noise robustness of the recently proposed Efficient Conformer Connectionist Temporal Classification (CTC)-based architecture by processing both audio and visual modalities. We improve previous lip reading methods using an Efficient Conformer back-end on top of a ResNet-18 visual front-end and by adding intermediate CTC losses between blocks. We condition intermediate block features on early predictions using Inter CTC residual modules to relax the conditional independence assumption of CTC-based models. We also replace the Efficient Conformer grouped attention by a more efficient and simpler attention mechanism that we call patch attention. We experiment with publicly available Lip Reading Sentences 2 (LRS2) and Lip Reading Sentences 3 (LRS3) datasets. Our experiments show that using audio and visual modalities allows to better recognize speech in the presence of environmental noise and significantly accelerate training, reaching lower WER with 4 times less training steps. Our Audio-Visual Efficient Conformer (AVEC) model achieves state-of-the-art performance, reaching WER of 2.3% and 1.8% on LRS2 and LRS3 test sets. Code and pretrained models are available at https://github.com/burchim/AVEC.

Continual Learning for On-Device Speech Recognition using Disentangled Conformers

Conformer-1: Robust ASR via Large-Scale Semisupervised Bootstrapping

CL-MASR: A Continual Learning Benchmark for Multilingual ASR

Audio-Visual Efficient Conformer for Robust Speech Recognition

Sharing Low Rank Conformer Weights for Tiny Always-On Ambient Speech Recognition Models

Efficient End-to-End Speech Recognition Using Performers in Conformers

A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Tasks

Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition

Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems

Continual Learning for End-to-End ASR by Averaging Domain Experts

Self-consistent context aware conformer transducer for speech recognition

Towards A Unified Conformer Structure: from ASR to ASV Task

FedSpeech: Federated Text-to-Speech with Continual Learning

Efficient conformer-based speech recognition with linear attention

Locality Matters: A Locality-Biased Linear Attention for Automatic Speech Recognition

Conformers are All You Need for Visual Speech Recognition

Federated Self-Learning with Weak Supervision for Speech Recognition

Continuously Learning New Words in Automatic Speech Recognition

Conformer-Based Speech Recognition On Extreme Edge-Computing Devices

Efficient and Robust Long-Form Speech Recognition with Hybrid H3-Conformer

Nextformer: A ConvNeXt Augmented Conformer For End-To-End Speech Recognition