Abstract:Speech is the most used communication method between humans and it involves the perception of auditory and visual channels. Automatic speech recognition focuses on interpreting the audio signals, although the video can provide information that is complementary to the audio. Exploiting the visual information, however, has proven challenging. On one hand, researchers have reported that the mapping between phonemes and visemes (visual units) is one-to-many because there are phonemes which are visually similar and indistinguishable between them. On the other hand, it is known that some people are very good lip-readers (e.g: deaf people). We study the limit of visual only speech recognition in controlled conditions. With this goal, we designed a new database in which the speakers are aware of being read and aim to facilitate lip-reading. In the literature, there are discrepancies on whether hearing-impaired people are better lip-readers than normal-hearing people. Then, we analyze if there are differences between the lip-reading abilities of 9 hearing-impaired and 15 normal-hearing people. Finally, human abilities are compared with the performance of a visual automatic speech recognition system. In our tests, hearing-impaired participants outperformed the normal-hearing participants but without reaching statistical significance. Human observers were able to decode 44% of the spoken message. In contrast, the visual only automatic system achieved 20% of word recognition rate. However, if we repeat the comparison in terms of phonemes both obtained very similar recognition rates, just above 50%. This suggests that the gap between human lip-reading and automatic speech-reading might be more related to the use of context than to the ability to interpret mouth appearance.

Lip Localization and Viseme Classification for Visual Speech Recognition

Visual Words for Automatic Lip-Reading

An arabic visual speech recognition framework with CNN and vision transformers for lipreading

An automatic lip reading for short sentences using deep learning nets

A Study on Lip Localization Techniques used for Lip reading from a Video

Automatic lip-reading classification using deep learning approaches and optimized quaternion meixner moments by GWO algorithm

Automatic Viseme Vocabulary Construction to Enhance Continuous Lip-reading

Cross-Attention Fusion of Visual and Geometric Features for Large Vocabulary Arabic Lipreading

Lip2Vec: Efficient and Robust Visual Speech Recognition via Latent-to-Latent Visual to Audio Representation Mapping

Lip reading using external viseme decoding

Lip-Listening: Mixing Senses to Understand Lips using Cross Modality Knowledge Distillation for Word-Based Models

Intuitive Perception - Speech Recognition using Machine Learning

Spatio-Temporal Attention Mechanism and Knowledge Distillation for Lip Reading

Leveraging Visemes for Better Visual Speech Representation and Lip Reading

Audio-Visual System for Robust Speaker Recognition.

Lipper: Synthesizing Thy Speech using Multi-View Lipreading

Electromyogram-Based Lip-Reading via Unobtrusive Dry Electrodes and Machine Learning Methods.

Language identification as improvement for lip-based biometric visual systems

Towards Estimating the Upper Bound of Visual-Speech Recognition: The Visual Lip-Reading Feasibility Database

Understanding the visual speech signal

Silenttalk: Lip Reading Through Ultrasonic Sensing on Mobile Phones