Abstract:To prevent users’ privacy from leakage, more and more mobile devices employ biometric-based authentication approaches, such as fingerprint, face recognition, voiceprint authentications, and so on, to enhance the privacy protection. However, these approaches are vulnerable to replay attacks. Although the state-of-art solutions utilize liveness verification to combat the attacks, existing approaches are sensitive to ambient environments, such as ambient lights and surrounding audible noises. Toward this end, we explore liveness verification of user authentication leveraging users’ mouth movements, which are robust to noisy environments. In this paper, we propose a lip reading-based user authentication system, LipPass, which extracts unique behavioral characteristics of users’ speaking mouths through acoustic sensing on smartphones for user authentication. We first investigate Doppler profiles of acoustic signals caused by users’ speaking mouths and find that there are unique mouth movement patterns for different individuals. To characterize the mouth movements, we propose a deep learning-based method to extract efficient features from Doppler profiles and employ softmax function, support vector machine, and support vector domain description to construct multi-class identifier, binary classifiers, and spoofer detectors for mouth state identification, user identification, and spoofer detection, respectively. Afterward, we develop a balanced binary tree-based authentication approach to accurately identify each individual leveraging these binary classifiers and spoofer detectors with respect to registered users. Through extensive experiments involving 48 volunteers in four real environments, LipPass can achieve 90.2% accuracy in user identification and 93.1% accuracy in spoofer detection.

Microwave Lip Reading Based on Programmable Metasurface Using Deep Learning

Microwave Lip Reading of Chinese Mandarin Based on Programmable Metasurface

Machine-learning Reprogrammable Metasurface Imager.

Microwave Speech Recognizer Empowered by a Programmable Metasurface

Lip Reading Based on 3D Face Modeling and Spatial Transformation Learning

Programmable Metasurface Based Microwave Gesture Detection and Recognition Using Deep Learning

Deep Learning-driven Adaptive Metasurface Transmission Focusing

Lip Reading-Based User Authentication Through Acoustic Sensing on Smartphones.

Microwave Reconstruction of 3D Human Facial Landmarks Using a Programmable Metasurface

Decoding lip language using triboelectric sensors with deep learning

Application of deep learning in Mandarin Chinese lip-reading recognition

Metamaterials Based Intelligent Microwave Human Behavior Recognition

Pushing the limits of remote RF sensing by reading lips under the face mask

HearMe: Accurate and Real-time Lip Reading based on Commercial RFID Devices

On the Application of Deep Learning in Modeling Metasurface

A Smart Metasurface for Electromagnetic Manipulation Based on Speech Recognition

Electromyogram-Based Lip-Reading via Unobtrusive Dry Electrodes and Machine Learning Methods.

Programmable Metasurface Based Microwave Biometric Identification

RF-Mic: Live Voice Eavesdropping Via Capturing Subtle Facial Speech Dynamics Leveraging RFID

LipFormer: Learning to Lipread Unseen Speakers based on Visual-Landmark Transformers

Deep-learning-enabled Self-Adaptive Microwave Cloak Without Human Intervention