Abstract:This article presents EARFace , a system that shows the feasibility of tracking facial landmarks for 3D facial reconstruction using in-ear acoustic sensors embedded within smart earphones. This enables a number of applications in the areas of facial expression tracking, user interfaces, AR/VR applications, affective computing, and accessibility, among others. Although conventional vision-based solutions break down under poor lighting and occlusions, and also suffer from privacy concerns, earphone platforms are robust to ambient conditions while being privacy-preserving. In contrast to prior work on earable platforms that perform outer-ear sensing for facial motion tracking, EARFace shows the feasibility of completely in-ear sensing with a natural earphone form factor, thus enhancing the comfort levels of wearing. The core intuition exploited by EARFace is that the shape of the ear canal changes due to the movement of facial muscles during facial motion. EARFace tracks the changes in shape of the ear canal by measuring ultrasonic channel frequency response of the inner ear, ultimately resulting in tracking of the facial motion. A transformer-based machine learning model is designed to exploit spectral and temporal relationships in the ultrasonic channel frequency response data to predict the facial landmarks of the user with an accuracy of 1.83 mm. Using these predicted landmarks, a 3D graphical model of the face that replicates the precise facial motion of the user is then reconstructed. Domain adaptation is further performed by adapting the weights of layers using a group-wise and differential learning rate. This decreases the training overhead in EARFace . The transformer-based machine learning model runs on smart phone devices with a processing latency of 13 ms and an overall low power consumption profile. Finally, usability studies indicate higher levels of comforts of wearing EARFace ’s earphone platform in comparison with alternative form factors.

EarphoneTrack: involving earphones into the ecosystem of acoustic motion tracking

Mmear: Push the Limit of COTS Mmwave Eavesdropping on Headphones

EHTrack: Earphone-Based Head Tracking via Only Acoustic Signals

I Am an Earphone and I Can Hear My User’s Face: Facial Landmark Tracking Using Smart Earphones

MagSound: Magnetic Field Assisted Wireless Earphone Tracking.

A Wearable Vision-To-Audio Sensory Substitution Device for Blind Assistance and the Correlated Neural Substrates

Poster: Inaudible High-throughput Communication Through Acoustic Signals

EarIO: A Low-power Acoustic Sensing Earable for Continuously Tracking Detailed Facial Movements.

FaceOri: Tracking Head Position and Orientation Using Ultrasonic Ranging on Earphones

Addressing Practical Challenges in Acoustic Sensing to Enable Fast Motion Tracking

MagSound

ReflecTrack: Enabling 3D Acoustic Position Tracking Using Commodity Dual-Microphone Smartphones

Robust Inertial Motion Tracking Through Deep Sensor Fusion Across Smart Earbuds and Smartphone.

FusionTrack: Towards Accurate Device-free Acoustic Motion Tracking with Signal Fusion

EarEcho

Real-Time and Accurate Acoustic Indoor Localization with A Smartphone

MilliSonic: Pushing the Limits of Acoustic Motion Tracking

Bo-ear: Unsupervised Car Sound Sensing and Tracking Using Microphones on Smartphones.

Underwater Target Tracking in Three Dimensional Space Based on Sound Speed Profile

Room-Scale Location Trace Tracking via Continuous Acoustic Waves

Vernier: Accurate and Fast Acoustic Motion Tracking Using Mobile Devices