Abstract:To prevent users’ privacy from leakage, more and more mobile devices employ biometric-based authentication approaches, such as fingerprint, face recognition, voiceprint authentications, and so on, to enhance the privacy protection. However, these approaches are vulnerable to replay attacks. Although the state-of-art solutions utilize liveness verification to combat the attacks, existing approaches are sensitive to ambient environments, such as ambient lights and surrounding audible noises. Toward this end, we explore liveness verification of user authentication leveraging users’ mouth movements, which are robust to noisy environments. In this paper, we propose a lip reading-based user authentication system, LipPass, which extracts unique behavioral characteristics of users’ speaking mouths through acoustic sensing on smartphones for user authentication. We first investigate Doppler profiles of acoustic signals caused by users’ speaking mouths and find that there are unique mouth movement patterns for different individuals. To characterize the mouth movements, we propose a deep learning-based method to extract efficient features from Doppler profiles and employ softmax function, support vector machine, and support vector domain description to construct multi-class identifier, binary classifiers, and spoofer detectors for mouth state identification, user identification, and spoofer detection, respectively. Afterward, we develop a balanced binary tree-based authentication approach to accurately identify each individual leveraging these binary classifiers and spoofer detectors with respect to registered users. Through extensive experiments involving 48 volunteers in four real environments, LipPass can achieve 90.2% accuracy in user identification and 93.1% accuracy in spoofer detection.

Texture-Constrained Shape Prediction for Mouth Contour Extraction and its State Estimation

Active shape model based on sparse representation

Facial Features Extraction Based on Active Shape Model.

Robust Three-step Facial Landmark Localization under the Complicated Condition via ASM and POEM.

Facial Feature Localization Using Robust Active Shape Model and POEM Descriptors

Improved Active Shape Model for Facial Feature Localization Using Poem Descriptor

Lip Reading-Based User Authentication Through Acoustic Sensing on Smartphones.

Robust Real-Time Face Alignment Based on ASM with Boosting Regression for Displacement Prediction

Hands-free Robot Control System Using Mouth Tracking

An Efficient Approach to Chinese Phoneme Mouth-Shape Recognition

Automatic tongue location and segmentation

Extraction and Classification of Mouth Shape Features in Oral English Teaching Based on Image Processing

Combining Active Shape Models and Active Appearance Models For Accurate Image Interpretation

An Improved Active Shape Model for Face Alignment

Enhance ASMs Based on AdaBoost-Based Salient Landmarks Localization and Confidence-Constraint Shape Modeling

Mouth-Shape Classification And Recognition For Lipreading

Automatic facial expression recognition on a single 3D face by exploring shape deformation.

Accurate and real-time lip contour extraction based on constrained contour growing

A Composite Method to Extract Eye Contour

Face alignment under various poses and expressions

Lip Contour Extraction Based on Support Vector Machine Add Support