Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera

Jiye Lee,Hanbyul Joo

2024-05-06

Abstract:We present a lightweight and affordable motion capture method based on two smartwatches and a head-mounted camera. In contrast to the existing approaches that use six or more expert-level IMU devices, our approach is much more cost-effective and convenient. Our method can make wearable motion capture accessible to everyone everywhere, enabling 3D full-body motion capture in diverse environments. As a key idea to overcome the extreme sparsity and ambiguities of sensor inputs with different modalities, we integrate 6D head poses obtained from the head-mounted cameras for motion estimation. To enable capture in expansive indoor and outdoor scenes, we propose an algorithm to track and update floor level changes to define head poses, coupled with a multi-stage Transformer-based regression module. We also introduce novel strategies leveraging visual cues of egocentric images to further enhance the motion capture quality while reducing ambiguities. We demonstrate the performance of our method on various challenging scenarios, including complex outdoor environments and everyday motions including object interactions and social interactions among multiple individuals.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The paper attempts to address the problem of achieving a lightweight and cost-effective motion capture method without the need for specialized equipment and complex setups. Specifically, the study utilizes smartwatches on both wrists and a head-mounted camera to achieve high-quality 3D full-body motion capture. Compared to existing methods that require multiple expert-level Inertial Measurement Unit (IMU) devices, this approach is more economical and user-friendly. Additionally, the paper proposes an algorithm to track and update floor height changes to define head posture and introduces a multi-stage Transformer-based regression module to enhance the quality of motion estimation. This method makes wearable motion capture possible in various environments, including indoor and outdoor settings, and can handle complex daily activities such as object interactions and social interactions. In summary, the study aims to democratize motion capture technology by popularizing easily accessible devices.