Abstract:This article presents EARFace , a system that shows the feasibility of tracking facial landmarks for 3D facial reconstruction using in-ear acoustic sensors embedded within smart earphones. This enables a number of applications in the areas of facial expression tracking, user interfaces, AR/VR applications, affective computing, and accessibility, among others. Although conventional vision-based solutions break down under poor lighting and occlusions, and also suffer from privacy concerns, earphone platforms are robust to ambient conditions while being privacy-preserving. In contrast to prior work on earable platforms that perform outer-ear sensing for facial motion tracking, EARFace shows the feasibility of completely in-ear sensing with a natural earphone form factor, thus enhancing the comfort levels of wearing. The core intuition exploited by EARFace is that the shape of the ear canal changes due to the movement of facial muscles during facial motion. EARFace tracks the changes in shape of the ear canal by measuring ultrasonic channel frequency response of the inner ear, ultimately resulting in tracking of the facial motion. A transformer-based machine learning model is designed to exploit spectral and temporal relationships in the ultrasonic channel frequency response data to predict the facial landmarks of the user with an accuracy of 1.83 mm. Using these predicted landmarks, a 3D graphical model of the face that replicates the precise facial motion of the user is then reconstructed. Domain adaptation is further performed by adapting the weights of layers using a group-wise and differential learning rate. This decreases the training overhead in EARFace . The transformer-based machine learning model runs on smart phone devices with a processing latency of 13 ms and an overall low power consumption profile. Finally, usability studies indicate higher levels of comforts of wearing EARFace ’s earphone platform in comparison with alternative form factors.

Combining IMU with Acoustics for Head Motion Tracking Leveraging Wireless Earphone

HeadTrack: Real-time Human-Computer Interaction Via Wireless Earphones

EarphoneTrack: involving earphones into the ecosystem of acoustic motion tracking

EHTrack: Earphone-Based Head Tracking via Only Acoustic Signals

Robust Inertial Motion Tracking Through Deep Sensor Fusion Across Smart Earbuds and Smartphone.

A Surveillance System for Drone Localization and Tracking Using Acoustic Arrays

MagSound: Magnetic Field Assisted Wireless Earphone Tracking.

FM-track: Pushing the Limits of Contactless Multi-Target Tracking Using Acoustic Signals

FusionTrack: Towards Accurate Device-free Acoustic Motion Tracking with Signal Fusion

Addressing Practical Challenges in Acoustic Sensing to Enable Fast Motion Tracking

FaceOri: Tracking Head Position and Orientation Using Ultrasonic Ranging on Earphones

HeadTracker: Fine-Grained Head Orientation Tracking System Based on Headphones.

Audio-Visual Bimodal Combination-Based Speaker Tracking Method for Mobile Robot

MagSound

CTrack: Acoustic Device-Free and Collaborative Hands Motion Tracking on Smartphones

Infrastructure-Free Indoor Pedestrian Tracking With Smartphone Acoustic-Based Enhancement

I Am an Earphone and I Can Hear My User’s Face: Facial Landmark Tracking Using Smart Earphones

Wi-Tracker: Monitoring Breathing Airflow with Acoustic Signals

PmTrack: Enabling Personalized mmWave-based Human Tracking

IndoTrack

Real-Time and Accurate Acoustic Indoor Localization with A Smartphone