emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation

Sasha Salter,Richard Warren,Collin Schlager,Adrian Spurr,Shangchen Han,Rohin Bhasin,Yujun Cai,Peter Walkington,Anuoluwapo Bolarinwa,Robert Wang,Nathan Danielson,Josh Merel,Eftychios Pnevmatikakis,Jesse Marshall

2024-12-03

Abstract:Hands are the primary means through which humans interact with the world. Reliable and always-available hand pose inference could yield new and intuitive control schemes for human-computer interactions, particularly in virtual and augmented reality. Computer vision is effective but requires one or multiple cameras and can struggle with occlusions, limited field of view, and poor lighting. Wearable wrist-based surface electromyography (sEMG) presents a promising alternative as an always-available modality sensing muscle activities that drive hand motion. However, sEMG signals are strongly dependent on user anatomy and sensor placement, and existing sEMG models have required hundreds of users and device placements to effectively generalize. To facilitate progress on sEMG pose inference, we introduce the emg2pose benchmark, the largest publicly available dataset of high-quality hand pose labels and wrist sEMG recordings. emg2pose contains 2kHz, 16 channel sEMG and pose labels from a 26-camera motion capture rig for 193 users, 370 hours, and 29 stages with diverse gestures - a scale comparable to vision-based hand pose datasets. We provide competitive baselines and challenging tasks evaluating real-world generalization scenarios: held-out users, sensor placements, and stages. emg2pose provides the machine learning community a platform for exploring complex generalization problems, holding potential to significantly enhance the development of sEMG-based human-computer interactions.

Computer Vision and Pattern Recognition,Human-Computer Interaction,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to develop a large and diverse benchmark dataset and model to promote the research on estimating hand postures from surface electromyography (sEMG) signals. Specifically, the paper aims to:

1. **Overcome the limitations of existing sEMG datasets**: Existing sEMG datasets are relatively small in scale, lacking sufficient number of users, diversity in sensor placement, and types of hand gestures, which limit the generalization ability of models.

2. **Provide a high - quality dataset**: In order to train more general sEMG - to - hand - posture models, a large - scale, high - quality dataset is required, which contains high - resolution sEMG signals from different users and accurate hand - posture labels.

3. **Address the complexity and variability of sEMG signals**: sEMG signals are greatly affected by users' anatomical structures and sensor positions. Therefore, a model that can handle these variations and generalize to new users and new scenarios is needed.

4. **Explore complex generalization problems**: Evaluate the performance of models in the face of unseen users, sensor placement methods, and types of hand gestures to ensure the robustness and reliability of models in practical applications.

To this end, the paper introduces the **emg2pose** dataset and benchmarking platform. This dataset contains 16 - channel, 2 - kHz sEMG signals and hand - posture labels from 193 users, with a duration of over 370 hours. In addition, the paper also provides three competitive baseline models and designs challenging tasks to evaluate the generalization ability of models in the real world.

### Main contributions
- **Large - scale dataset**: emg2pose is currently the largest publicly available sEMG hand - posture dataset, covering diverse types of hand gestures.
- **High - quality labels**: High - precision hand - posture labels are obtained through a motion - capture system with 26 cameras.
- **Challenging tasks**: Generalization tasks for unseen users, sensor placement methods, and types of hand gestures are designed.
- **Baseline models**: Three different baseline models (vemg2pose, NeuroPose, and SensingDynamics) are provided for researchers to compare and improve.

Through these contributions, the paper provides researchers with a powerful tool to promote the further development of sEMG in the field of hand - posture estimation, especially in virtual reality (VR), augmented reality (AR), and other human - computer interaction applications.