Leveraging Visemes for Better Visual Speech Representation and Lip Reading

Javad Peymanfard,Vahid Saeedi,Mohammad Reza Mohammadi,Hossein Zeinali,Nasser Mozayani

2023-07-20

Abstract:Lip reading is a challenging task that has many potential applications in speech recognition, human-computer interaction, and security systems. However, existing lip reading systems often suffer from low accuracy due to the limitations of video features. In this paper, we propose a novel approach that leverages visemes, which are groups of phonetically similar lip shapes, to extract more discriminative and robust video features for lip reading. We evaluate our approach on various tasks, including word-level and sentence-level lip reading, and audiovisual speech recognition using the Arman-AV dataset, a largescale Persian corpus. Our experimental results show that our viseme based approach consistently outperforms the state-of-theart methods in all these tasks. The proposed method reduces the lip-reading word error rate (WER) by 9.1% relative to the best previous method.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to address the issue of improving accuracy in lip reading (visual speech recognition) tasks. Existing lip reading systems often have low accuracy due to the limitations of video features. This paper proposes a new method that utilizes visemes, which are groups of lip shapes that are similar in speech, to extract more distinctive and robust video features, thereby improving lip reading technology. The authors evaluated their method through various tasks, including word-level and sentence-level lip reading, as well as audiovisual speech recognition, using the large-scale Persian corpus Arman-A V. Experimental results show that the viseme-based method outperforms the current state-of-the-art methods in all these tasks, particularly reducing the word error rate (WER) of lip reading by 9.1% relative.