Abstract:Lip reading has received an increasing research interest in recent years due to the rapid development of deep learning and its widespread potential applications. One key point to obtain good performance for the lip reading task depends heavily on how effective the representation can be to capture the lip movement information and meanwhile to resist the noises resulted from the change of pose, lighting conditions, speaker's appearance and so on. Towards this target, we propose to introduce the mutual information constraints on both the local feature's level and the global sequence's level to enhance the relations of the features with the speech content. On the one hand, we constraint the features generated at each time step to enable them carry a strong relation with the speech content by imposing the local mutual information maximization constraint (LMIM), leading to improvements over the model's ability to discover fine-grained lip movements and the fine-grained differences among words with similar pronunciation, such as ``spend'' and ``spending''. On the other hand, we introduce the mutual information maximization constraint on the global sequence's level (GMIM), to make the model be able to pay more attention to discriminate key frames related with the speech content, and less to various noises appeared in the speaking process. By combining these two advantages together, the proposed method is expected to be both discriminative and robust for effective lip reading. To verify this method, we evaluate on two large-scale benchmark. We perform a detailed analysis and comparison on several aspects, including the comparison of the LMIM and GMIM with the baseline, the visualization of the learned representation and so on. The results not only prove the effectiveness of the proposed method but also report new state-of-the-art performance on both the two benchmarks.

A Lipreading Model Based on Fine-Grained Global Synergy of Lip Movement

Learn an Effective Lip Reading Model without Pains

Lip Reading Based on 3D Face Modeling and Spatial Transformation Learning

Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers

Multi-Grained Spatio-temporal Modeling for Lip-reading

LipFormer: Learning to Lipread Unseen Speakers based on Visual-Landmark Transformers

Combining Global and Local Classifiers for Lipreading

Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge

Mutual Information Maximization for Effective Lip Reading

Synchronous Bidirectional Learning for Multilingual Lip Reading

LCSNet: End-to-End Lipreading with Channel-aware Feature Selection

Generalizing sentence-level lipreading to unseen speakers: a two-stream end-to-end approach

Cross-Modal Language Modeling in Multi-Motion-Informed Context for Lip Reading

Landmark-Guided Cross-Speaker Lip Reading with Mutual Information Regularization

Video Analysis Using Spatiotemporal Descriptor and Kernel Extreme Learning Machine for Lip Reading

Learning the Relative Dynamic Features for Word-Level Lipreading

A Lip-Reading Recognition Approach Based on Long Short-Term Memory

Joint Geometry and Gray-Level Histogram Model for Lip-Reading

Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language

Lip Feature Analyzing in Speech Synthesis System for Speech Impaired

Spatio-Temporal Fusion Based Convolutional Sequence Learning for Lip Reading