Abstract:Background Model-based 3D pose estimation has been widely used in many 3D human motion analysis applications, in which vision-based and inertial-based are two distinct lines. Multi-view images in a vision-based markerless capture system provide essential data for motion analysis, but erroneous estimates still occur due to ambiguities, occlusion, or noise in images. Besides, the multi-view setting is hard for the application in the wild. Although inertial measurement units (IMUs) can obtain accurate direction without occlusion, they are usually susceptible to magnetic field interference and drifts. Hybrid motion capture has drawn the attention of researchers in recent years. Existing 3D pose estimation methods jointly optimize the parameters of the 3D pose by minimizing the discrepancy between the image and IMU data. However, these hybrid methods still suffer from the issues such as complex peripheral devices, sensitivity to initialization, and slow convergence. Methods This article presents an approach to improve 3D human pose estimation by fusing a single image with sparse inertial measurement units (IMUs). Based on a dual-stream feature extract network, we design a model-attention network with a residual module to closely couple the dual-modal feature from a static image and sparse inertial measurement units. The final 3D pose and shape parameters are directly obtained by a regression strategy. Results Extensive experiments are conducted on two benchmark datasets for 3D human pose estimation. Compared to state-of-the-art methods, the per vertex error (PVE) of human mesh reduces by 9.4 mm on Total Capture dataset and the mean per joint position error (MPJPE) reduces by 7.8 mm on the Human3.6M dataset. The quantitative comparison demonstrates that the proposed method could effectively fuse sparse IMU data and images and improve pose accuracy.

DeepIM : Deep Iterative Matching for 6 D Pose Estimation-Supplementary Material

DeepIM: Deep Iterative Matching for 6D Pose Estimation

3D Point-to-Keypoint Voting Network for 6D Pose Estimation

DeepRM: Deep Recurrent Matching for 6D Pose Refinement

6IMPOSE: bridging the reality gap in 6D pose estimation for robotic grasping

Deep Learning-Based 6-DoF Object Pose Estimation Considering Synthetic Dataset

FEIF: Feature Excitation and Interactive Fusion for 6D Object Pose Estimation.

Mask6D: Masked Pose Priors for 6D Object Pose Estimation.

PoseMatcher: One-shot 6D Object Pose Estimation by Deep Feature Matching

StablePose: Learning 6D Object Poses from Geometrically Stable Patches

End-to-End Implicit Object Pose Estimation

6-D Object Pose Estimation Based on Point Pair Matching for Robotic Grasp Detection

A Deconvolutional Bottom-up Deep Network for Multi-Person Pose Estimation.

SimMIM: A Simple Framework for Masked Image Modeling

Deep instance segmentation and 6D object pose estimation in cluttered scenes for robotic autonomous grasping

DPPMask: Masked Image Modeling with Determinantal Point Processes

SuperPose: Improved 6D Pose Estimation with Robust Tracking and Mask-Free Initialization

SO-Pose: Exploiting Self-Occlusion for Direct 6D Pose Estimation

Reconstructing 3D human pose and shape from a single image and sparse IMUs

A Learning-based Variable Size Part Extraction Architecture for 6D Object Pose Recovery in Depth

A modal fusion network with dual attention mechanism for 6D pose estimation