Abstract:Wearable sensors such as Inertial Measurement Units (IMUs) are often used to assess the performance of human exercise. Common approaches use handcrafted features based on domain expertise or automatically extracted features using time series analysis. Multiple sensors are required to achieve high classification accuracy, which is not very practical. These sensors require calibration and synchronization and may lead to discomfort over longer time periods. Recent work utilizing computer vision techniques has shown similar performance using video, without the need for manual feature engineering, and avoiding some pitfalls such as sensor calibration and placement on the body. In this paper, we compare the performance of IMUs to a video-based approach for human exercise classification on two real-world datasets consisting of Military Press and Rowing exercises. We compare the performance using a single camera that captures video in the frontal view versus using 5 IMUs placed on different parts of the body. We observe that an approach based on a single camera can outperform a single IMU by 10 percentage points on average. Additionally, a minimum of 3 IMUs are required to outperform a single camera. We observe that working with the raw data using multivariate time series classifiers outperforms traditional approaches based on handcrafted or automatically extracted features. Finally, we show that an ensemble model combining the data from a single camera with a single IMU outperforms either data modality. Our work opens up new and more realistic avenues for this application, where a video captured using a readily available smartphone camera, combined with a single sensor, can be used for effective human exercise classification.

Synthetic Smartwatch IMU Data Generation from In-the-wild ASL Videos

Enhancing Inertial Hand based HAR through Joint Representation of Language, Pose and Synthetic IMUs

Video2IMU: Realistic IMU features and signals from videos

A Scalable and Wearable Self-Sensing IMU Sensor Network for Personalized Human Motion and Deformation Capture

WristCam: A Wearable Sensor for Hand Trajectory Gesture Recognition and Intelligent Human–Robot Interaction

Robust Finger Interactions with COTS Smartwatches via Unsupervised Siamese Adaptation

Gesture Recognition with a 3-D Accelerometer

An Examination of Wearable Sensors and Video Data Capture for Human Exercise Classification

American Sign Language Translation Using Wearable Inertial and Electromyography Sensors for Tracking Hand Movements and Facial Expressions

RGC: Reliable Gesture Classification Via Wearables Using GANs-Based Data Augmentation.

ImAiR: Airwriting Recognition framework using Image Representation of IMU Signals

Sensor Fusion of Motion-Based Sign Language Interpretation with Deep Learning

Sensor Data Augmentation from Skeleton Pose Sequences for Improving Human Activity Recognition

Human activities recognition with a single writs IMU via a Variational Autoencoder and android deep recurrent neural nets

American society of biomechanics early career achievement award 2020: Toward portable and modular biomechanics labs: How video and IMU fusion will change gait analysis

Leveraging Sound and Wrist Motion to Detect Activities of Daily Living with Commodity Smartwatches

Ballroom Dance Movement Recognition Using a Smart Watch

Sign Language Recognition Analysis using Multimodal Data

IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition

A comparative study on wearables and single-camera video for upper-limb out-of-thelab activity recognition with different deep learning architectures

DiffSign: AI-Assisted Generation of Customizable Sign Language Videos With Enhanced Realism