Abstract:To prevent users’ privacy from leakage, more and more mobile devices employ biometric-based authentication approaches, such as fingerprint, face recognition, voiceprint authentications, and so on, to enhance the privacy protection. However, these approaches are vulnerable to replay attacks. Although the state-of-art solutions utilize liveness verification to combat the attacks, existing approaches are sensitive to ambient environments, such as ambient lights and surrounding audible noises. Toward this end, we explore liveness verification of user authentication leveraging users’ mouth movements, which are robust to noisy environments. In this paper, we propose a lip reading-based user authentication system, LipPass, which extracts unique behavioral characteristics of users’ speaking mouths through acoustic sensing on smartphones for user authentication. We first investigate Doppler profiles of acoustic signals caused by users’ speaking mouths and find that there are unique mouth movement patterns for different individuals. To characterize the mouth movements, we propose a deep learning-based method to extract efficient features from Doppler profiles and employ softmax function, support vector machine, and support vector domain description to construct multi-class identifier, binary classifiers, and spoofer detectors for mouth state identification, user identification, and spoofer detection, respectively. Afterward, we develop a balanced binary tree-based authentication approach to accurately identify each individual leveraging these binary classifiers and spoofer detectors with respect to registered users. Through extensive experiments involving 48 volunteers in four real environments, LipPass can achieve 90.2% accuracy in user identification and 93.1% accuracy in spoofer detection.

Electromyogram-Based Lip-Reading via Unobtrusive Dry Electrodes and Machine Learning Methods.

LipPass: Lip Reading-based User Authentication on Smartphones Leveraging Acoustic Signals.

Decoding lip language using triboelectric sensors with deep learning

Lip Reading-Based User Authentication Through Acoustic Sensing on Smartphones.

Speech neuromuscular decoding based on spectrogram images using conformal predictors with Bi-LSTM.

Lip Reading Based on 3D Face Modeling and Spatial Transformation Learning

A data-efficient and easy-to-use lip language interface based on wearable motion capture and speech movement reconstruction

Decoding Silent Speech Commands from Articulatory Movements Through Soft Magnetic Skin and Machine Learning

HearMe: Accurate and Real-time Lip Reading based on Commercial RFID Devices

Acoustic-Based Lip Reading for Mobile Devices: Dataset, Benchmark and a Self Distillation-Based Approach

Silenttalk: Lip Reading Through Ultrasonic Sensing on Mobile Phones

Pushing the limits of remote RF sensing by reading lips under the face mask

Laser-Induced Graphene Strain Sensor for Conformable Lip-Reading Recognition and Human–Machine Interaction

End-to-end Neuromorphic Lip Reading

Advances and Challenges in Deep Lip Reading

An epidermal sEMG tattoo-like patch as a new human-machine interface for patients with loss of voice

Towards EMG-to-Speech with a Necklace Form Factor

EMOLIPS: Towards Reliable Emotional Speech Lip-Reading

Stretchable and durable HD-sEMG electrodes for accurate recognition of swallowing activities on complex epidermal surfaces

Lipper: Synthesizing Thy Speech using Multi-View Lipreading

Visual Words for Automatic Lip-Reading