Abstract:Background Model-based 3D pose estimation has been widely used in many 3D human motion analysis applications, in which vision-based and inertial-based are two distinct lines. Multi-view images in a vision-based markerless capture system provide essential data for motion analysis, but erroneous estimates still occur due to ambiguities, occlusion, or noise in images. Besides, the multi-view setting is hard for the application in the wild. Although inertial measurement units (IMUs) can obtain accurate direction without occlusion, they are usually susceptible to magnetic field interference and drifts. Hybrid motion capture has drawn the attention of researchers in recent years. Existing 3D pose estimation methods jointly optimize the parameters of the 3D pose by minimizing the discrepancy between the image and IMU data. However, these hybrid methods still suffer from the issues such as complex peripheral devices, sensitivity to initialization, and slow convergence. Methods This article presents an approach to improve 3D human pose estimation by fusing a single image with sparse inertial measurement units (IMUs). Based on a dual-stream feature extract network, we design a model-attention network with a residual module to closely couple the dual-modal feature from a static image and sparse inertial measurement units. The final 3D pose and shape parameters are directly obtained by a regression strategy. Results Extensive experiments are conducted on two benchmark datasets for 3D human pose estimation. Compared to state-of-the-art methods, the per vertex error (PVE) of human mesh reduces by 9.4 mm on Total Capture dataset and the mean per joint position error (MPJPE) reduces by 7.8 mm on the Human3.6M dataset. The quantitative comparison demonstrates that the proposed method could effectively fuse sparse IMU data and images and improve pose accuracy.

Pose Determination from Multi-View Image Using Deep Learning

Modelling Human Body Pose for Action Recognition Using Deep Neural Networks

Multi-View Pose Generator Based on Deep Learning for Monocular 3D Human Pose Estimation

Combining Local Appearance and Holistic View: Dual-Source Deep Neural Networks for Human Pose Estimation

Pose Estimation for Non-Cooperative Spacecraft Rendezvous Using Convolutional Neural Networks

Recent Advances in Monocular 2D and 3D Human Pose Estimation: A Deep Learning Perspective

3d Pose Detection Of Closely Interactive Humans Using Multi-View Cameras

Human Pose Estimation from Depth Images via Inference Embedded Multi-task Learning

Fast and Robust Multi-Person 3D Pose Estimation from Multiple Views

Human Pose Estimation with Regression by Fusing Multi-View Visual Information

Deep Dual Consecutive Network for Human Pose Estimation

Multi-Human Pose Estimation by Deep Learning-Based Sequential Approach for Human Keypoint Position and Human Body Detection

Stereo Visual Odometry Pose Correction through Unsupervised Deep Learning

Spacecraft Homography Pose Estimation with Single-Stage Deep Convolutional Neural Network

A Two-Stage Monocular Vision Detection Method for 6D Pose Estimation in Multi-Heterogeneous Robot Systems

Human Pose Estimation Using Deep Structure Guided Learning.

Reconstructing 3D human pose and shape from a single image and sparse IMUs

Recognizing body poses using multilinear analysis and semi-supervised learning

GS-Pose: Category-Level Object Pose Estimation via Geometric and Semantic Correspondence

3D Human Pose Estimation from Deep Multi-View 2D Pose

Multi-Scale Supervised Network for Human Pose Estimation