Abstract:Smart ear-worn devices (called earables) are being equipped with various onboard sensors and algorithms, transforming earphones from simple audio transducers to multi-modal interfaces making rich inferences about human motion and vital signals. However, developing sensory applications using earables is currently quite cumbersome with several barriers in the way. First, time-series data from earable sensors incorporate information about physical phenomena in complex settings, requiring machine-learning (ML) models learned from large-scale labeled data. This is challenging in the context of earables because large-scale open-source datasets are missing. Secondly, the small size and compute constraints of earable devices make on-device integration of many existing algorithms for tasks such as human activity and head-pose estimation difficult. To address these challenges, we introduce Auritus an extendable and open-source optimization toolkit designed to enhance and replicate earable applications. Auritus serves two primary functions. Firstly, Auritus handles data collection, pre-processing, and labeling tasks for creating customized earable datasets using graphical tools. The system includes an open-source dataset with 2.43 million inertial samples related to head and full-body movements, consisting of 34 head poses and 9 activities from 45 volunteers. Secondly, Auritus provides a tightly-integrated hardware-in-the-loop (HIL) optimizer and TinyML interface to develop lightweight and real-time machine-learning (ML) models for activity detection and filters for head-pose tracking. To validate the utlity of Auritus, we showcase three sample applications, namely fall detection, spatial audio rendering, and augmented reality (AR) interfacing. Auritus recognizes activities with 91% leave 1-out test accuracy (98% test accuracy) using real-time models as small as 6-13 kB. Our models are 98-740× smaller and 3-6% more accurate over the state-of-the-art. We also estimate head pose with absolute errors as low as 5 degrees using 20kB filters, achieving up to 1.6× precision improvement over existing techniques. We make the entire system open-source so that researchers and developers can contribute to any layer of the system or rapidly prototype their applications using our dataset and algorithms.

The EarSAVAS Dataset: Enabling Subject-Aware Vocal Activity Sensing on Earables.

The EarSAVAS Dataset

EarDA: Towards Accurate and Data-Efficient Earable Activity Sensing

Human Activity Recognition in a Free-Living Environment Using an Ear-Worn Motion Sensor

SLearn: Shared Learning Human Activity Labels Across Multiple Datasets.

ActSonic: Recognizing Everyday Activities from Inaudible Acoustic Wave Around the Body

AudioEar: Single-View Ear Reconstruction for Personalized Spatial Audio

Continual learning in sensor-based human activity recognition: An empirical benchmark analysis

Aria Everyday Activities Dataset

Auritus: An Open-Source Optimization Toolkit for Training and Development of Human Movement Models and Filters Using Earables

EarSpeech: Exploring In-Ear Occlusion Effect on Earphones for Data-efficient Airborne Speech Enhancement

HeAR -- Health Acoustic Representations

EarCough: Enabling Continuous Subject Cough Event Detection on Hearables

COVID-19 Sounds: A Large-Scale Audio Dataset for Digital Respiratory Screening

Live Speech Recognition Via Earphone Motion Sensors

Multimodal Earable Sensing for Human Energy Expenditure Estimation

EarEcho

An evaluation of heart rate monitoring with in-ear microphones under motion

I Can Hear You Without a Microphone: Live Speech Eavesdropping from Earphone Motion Sensors.

MAVD: The First Open Large-Scale Mandarin Audio-Visual Dataset with Depth Information

EarSleep: In-ear Acoustic-based Physical and Physiological Activity Recognition for Sleep Stage Detection