Abstract:Background Model-based 3D pose estimation has been widely used in many 3D human motion analysis applications, in which vision-based and inertial-based are two distinct lines. Multi-view images in a vision-based markerless capture system provide essential data for motion analysis, but erroneous estimates still occur due to ambiguities, occlusion, or noise in images. Besides, the multi-view setting is hard for the application in the wild. Although inertial measurement units (IMUs) can obtain accurate direction without occlusion, they are usually susceptible to magnetic field interference and drifts. Hybrid motion capture has drawn the attention of researchers in recent years. Existing 3D pose estimation methods jointly optimize the parameters of the 3D pose by minimizing the discrepancy between the image and IMU data. However, these hybrid methods still suffer from the issues such as complex peripheral devices, sensitivity to initialization, and slow convergence. Methods This article presents an approach to improve 3D human pose estimation by fusing a single image with sparse inertial measurement units (IMUs). Based on a dual-stream feature extract network, we design a model-attention network with a residual module to closely couple the dual-modal feature from a static image and sparse inertial measurement units. The final 3D pose and shape parameters are directly obtained by a regression strategy. Results Extensive experiments are conducted on two benchmark datasets for 3D human pose estimation. Compared to state-of-the-art methods, the per vertex error (PVE) of human mesh reduces by 9.4 mm on Total Capture dataset and the mean per joint position error (MPJPE) reduces by 7.8 mm on the Human3.6M dataset. The quantitative comparison demonstrates that the proposed method could effectively fuse sparse IMU data and images and improve pose accuracy.

Fused Pose Estimation Using Geometric And Texture Information

Direct Alignment with Generalized Correspondences: A Unified Framework for Structure-Based Visual Pose Estimation.

Zero-Shot 3d Pose Estimation of Unseen Object by Two-Step Rgb-D Fusion

Fusion-Competition Framework of Local Topology and Global Texture for Head Pose Estimation

Image Alignment Via Multi-Model Geometric Fitting and Hierarchical Homography Estimation

Triangulate Geometric Constraint Combined with Visual-Flow Fusion Network for Accurate 6dof Pose Estimation.

Research on 3D Human Pose Estimation Technique Based on Multi-View Information Fusion

A Pose Estimation Algorithm for Multimodal Data Fusion

3D Human Pose Estimation Based on Multi View Information Fusion

Uncooperative Spacecraft Pose Estimation Based on Intensity and Range Images Fusion

Reconstructing 3D human pose and shape from a single image and sparse IMUs

Towards Locality Similarity Preserving to 3D Human Pose Estimation.

Relative Pose Estimation Algorithm with Gyroscope Sensor.

Pose Estimation for Non-Cooperative Targets Using 3D Feature Correspondences Grouped Via Local and Global Constraints

3D Hand Pose and Shape Estimation from Monocular RGB Via Efficient 2D Cues

Lifting by Image -- Leveraging Image Cues for Accurate 3D Human Pose Estimation

6D Pose Estimation Method Using Curvature-Enhanced Point-Pair Features

Hybrid Silhouette and Key-Point Driven 3D-2D Registration

Pose Estimation of Non-Cooperative Space Targets Based on Cross-Source Point Cloud Fusion

Geometric Pose Affordance: 3D Human Pose with Scene Constraints

Fast vision-based pose estimation iterative algorithm