Abstract:With the breakthrough of deep learning, lip reading technologies are under extraordinarily rapid progress. It is well-known that Chinese is the most widely spoken language in the world. Unlike alphabetic languages, it involves more than 1,000 pronunciations as Pinyin, and nearly 90,000 pictographic characters as Hanzi, which makes lip reading of Chinese very challenging. In this paper, we implement visual-only Chinese lip reading of unconstrained sentences in a two-step end-to-end architecture (LipCH-Net), in which two deep neural network models are employed to perform the recognition of Picture-to-Pinyin (mouth motion pictures to pronunciations) and the recognition of Pinyin-to-Hanzi (pronunciations to texts) respectively, before having a jointly optimization to improve the overall performance. In addition, two modules in the Pinyin-to-Hanzi model are pre-trained separately with large auxiliary data in advance of sequence-to-sequence training to make the best of long sequence matches for avoiding ambiguity. We collect 6-month daily news broadcasts from China Central Television (CCTV) website, and semi-automatically label them into a 20.95 GB dataset with 20,495 natural Chinese sentences. When trained on the CCTV dataset, the LipCH-Net model outperforms the performance of all state-of-the-art lip reading frameworks. According to the results, our scheme not only accelerates training and reduces over-fitting, but also overcomes syntactic ambiguity of Chinese which provides a baseline for future relevant work.

Lip Contour Recognition Based on Context Information of Chinese Phoneme

Improving Chinese Lip-Reading Recognizing Rate by Unsymmetrical Lip Contour Model

Lip Feature Analyzing in Speech Synthesis System for Speech Impaired

Feature Analysis of Lip's Image Sequences Based on Weighted Combination

Lip Contour Description Based on Orthogonal Transform in Visual Driven Speech Synthesis System

Chinese Tone Recognition Based on 3D Dynamic Muscle Information

A kind of improving HMM model and using in the visual speech recognition

A study on improved hidden Markov models and applications to speech recognition

Lip-Movement Features Extraction and Recognition Based on Chroma Analysis

Variable Structure and Modeling Units for Chinese Lipreading

Recognition of Sequence Lip Images and Its Application

Application of deep learning in Mandarin Chinese lip-reading recognition

Understanding Pictograph with Facial Features: End-to-end Sentence-Level Lip Reading of Chinese

HMM-based Lip Reading with Stingy Residual 3D Convolution

Lipreading HLM and Text Flow Analysis

Lip Reading Based on Cascade Feature Extraction and Hmm

Lip Feature Fusion in Speech Synthesis System Driven by Visual-Speech for Speech Impaired

Algorithm for Mandarin Continuous Speech Recognition Based on Context-Dependent Unit Between Syllables

A Lip-Reading Recognition Approach Based on Long Short-Term Memory

Lip Feature Extraction Based on Pulse Coupled Neural Network

Real Context Model for Tone Recognition in Mandarin Conversational Telephone Speech