Abstract:Lipreading is an impressive technique and there has been a definite improvement of accuracy in recent years. However, existing methods for lipreading mainly build on autoregressive (AR) model, which generate target tokens one by one and suffer from high inference latency. To breakthrough this constraint, we propose FastLR, a non-autoregressive (NAR) lipreading model which generates all target tokens simultaneously. NAR lipreading is a challenging task that has many difficulties: 1) the discrepancy of sequence lengths between source and target makes it difficult to estimate the length of the output sequence; 2) the conditionally independent behavior of NAR generation lacks the correlation across time which leads to a poor approximation of target distribution; 3) the feature representation ability of encoder can be weak due to lack of effective alignment mechanism; and 4) the removal of AR language model exacerbates the inherent ambiguity problem of lipreading. Thus, in this paper, we introduce three methods to reduce the gap between FastLR and AR model: 1) to address challenges 1 and 2, we leverage integrate-and-fire (I&F) module to model the correspondence between source video frames and output text sequence. 2) To tackle challenge 3, we add an auxiliary connectionist temporal classification (CTC) decoder to the top of the encoder and optimize it with extra CTC loss. We also add an auxiliary autoregressive decoder to help the feature extraction of encoder. 3) To overcome challenge 4, we propose a novel Noisy Parallel Decoding (NPD) for I&F and bring Byte-Pair Encoding (BPE) into lipreading. Our experiments exhibit that FastLR achieves the speedup up to 10.97× comparing with state-of-the-art lipreading model with slight WER absolute increase of 1.5% and 5.5% on GRID and LRS2 lipreading datasets respectively, which demonstrates the effectiveness of our proposed method.

Improved Word-level Lipreading with Temporal Shrinkage Network and NetVLAD

TCS-LipNet: Temporal & Channel & Spatial Attention-Based Lip Reading Network

Lip Reading Based on 3D Face Modeling and Spatial Transformation Learning

Learning the Relative Dynamic Features for Word-Level Lipreading

LCSNet: End-to-End Lipreading with Channel-aware Feature Selection

Multi-grained Spatio-Temporal Features Perceived Network for Event-based Lip-Reading

Application of deep learning in Mandarin Chinese lip-reading recognition

Lipreading Using Joint Preception Temporal Convolutional Network

Tackling Event-Based Lip-Reading by Exploring Multigrained Spatiotemporal Clues

FastLR

Deep Audio-visual System for Closed-set Word-level Speech Recognition

Utilizing Video Word Boundaries and Feature-Based Knowledge Distillation Improving Sentence-Level Lip Reading.

Research on a Lip Reading Algorithm Based on Efficient-GhostNet

Importance-Aware Information Bottleneck Learning Paradigm for Lip Reading

Efficient DNN Model for Word Lip-Reading

MALip: Modal Amplification Lipreading Based on Reconstructed Audio Features

Lipreading Architecture Based on Multiple Convolutional Neural Networks for Sentence-Level Visual Speech Recognition

Lip-reading with Densely Connected Temporal Convolutional Networks

HMM-based Lip Reading with Stingy Residual 3D Convolution

A Lip-Reading Recognition Approach Based on Long Short-Term Memory

Multi-Grained Spatio-temporal Modeling for Lip-reading