Abstract:Background Model-based 3D pose estimation has been widely used in many 3D human motion analysis applications, in which vision-based and inertial-based are two distinct lines. Multi-view images in a vision-based markerless capture system provide essential data for motion analysis, but erroneous estimates still occur due to ambiguities, occlusion, or noise in images. Besides, the multi-view setting is hard for the application in the wild. Although inertial measurement units (IMUs) can obtain accurate direction without occlusion, they are usually susceptible to magnetic field interference and drifts. Hybrid motion capture has drawn the attention of researchers in recent years. Existing 3D pose estimation methods jointly optimize the parameters of the 3D pose by minimizing the discrepancy between the image and IMU data. However, these hybrid methods still suffer from the issues such as complex peripheral devices, sensitivity to initialization, and slow convergence. Methods This article presents an approach to improve 3D human pose estimation by fusing a single image with sparse inertial measurement units (IMUs). Based on a dual-stream feature extract network, we design a model-attention network with a residual module to closely couple the dual-modal feature from a static image and sparse inertial measurement units. The final 3D pose and shape parameters are directly obtained by a regression strategy. Results Extensive experiments are conducted on two benchmark datasets for 3D human pose estimation. Compared to state-of-the-art methods, the per vertex error (PVE) of human mesh reduces by 9.4 mm on Total Capture dataset and the mean per joint position error (MPJPE) reduces by 7.8 mm on the Human3.6M dataset. The quantitative comparison demonstrates that the proposed method could effectively fuse sparse IMU data and images and improve pose accuracy.

Capturing the Unseen: Vision-Free Facial Motion Capture Using Inertial Measurement Units

HybridCap: Inertia-Aid Monocular Capture of Challenging Human Motions

Fusing Monocular Images and Sparse IMU Signals for Real-time Human Motion Capture

Motion Capture from Inertial and Vision Sensors

Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera

Imocap: Motion Capture from Internet Videos

DeepMultiCap: Performance Capture of Multiple Characters Using Sparse Multiview Cameras

HiFECap: Monocular High-Fidelity and Expressive Capture of Human Performances

MulayCap: Multi-layer Human Performance Capture Using A Monocular Video Camera

SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos

Towards Unstructured Unlabeled Optical Mocap: A Video Helps!

InMyFace: Inertial and mechanomyography-based sensor fusion for wearable facial activity recognition

Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera

IMU2Face: Real-time Gesture-driven Facial Reenactment

Hierarchical facial expression animation by motion capture data

Fusion Poser: 3D Human Pose Estimation Using Sparse IMUs and Head Trackers in Real Time

MVINS: Tightly-Coupled Mocap-Visual-Inertial Fusion for Global and Drift-Free Pose Estimation

ChallenCap: Monocular 3D Capture of Challenging Human Performances using Multi-Modal References

Reconstructing 3D human pose and shape from a single image and sparse IMUs

HybridFusion: Real-Time Performance Capture Using a Single Depth Sensor and Sparse IMUs

DeepMoCap: Deep Optical Motion Capture Using Multiple Depth Sensors and Retro-Reflectors