Abstract:Background Model-based 3D pose estimation has been widely used in many 3D human motion analysis applications, in which vision-based and inertial-based are two distinct lines. Multi-view images in a vision-based markerless capture system provide essential data for motion analysis, but erroneous estimates still occur due to ambiguities, occlusion, or noise in images. Besides, the multi-view setting is hard for the application in the wild. Although inertial measurement units (IMUs) can obtain accurate direction without occlusion, they are usually susceptible to magnetic field interference and drifts. Hybrid motion capture has drawn the attention of researchers in recent years. Existing 3D pose estimation methods jointly optimize the parameters of the 3D pose by minimizing the discrepancy between the image and IMU data. However, these hybrid methods still suffer from the issues such as complex peripheral devices, sensitivity to initialization, and slow convergence. Methods This article presents an approach to improve 3D human pose estimation by fusing a single image with sparse inertial measurement units (IMUs). Based on a dual-stream feature extract network, we design a model-attention network with a residual module to closely couple the dual-modal feature from a static image and sparse inertial measurement units. The final 3D pose and shape parameters are directly obtained by a regression strategy. Results Extensive experiments are conducted on two benchmark datasets for 3D human pose estimation. Compared to state-of-the-art methods, the per vertex error (PVE) of human mesh reduces by 9.4 mm on Total Capture dataset and the mean per joint position error (MPJPE) reduces by 7.8 mm on the Human3.6M dataset. The quantitative comparison demonstrates that the proposed method could effectively fuse sparse IMU data and images and improve pose accuracy.

High-Accuracy Pose Estimation for 3D Reconstruction Considering the Manipulator Motion Constraints with an Eye-in-Hand RGB-D Camera

Pose Estimation and Neural Implicit Reconstruction Towards Non-Cooperative Spacecraft Without Offline Prior Information

Motion Imitation of a Humanoid Robot Via Pose Estimation

In-Hand 3D Object Reconstruction from a Monocular RGB Video

Robust And Accurate Multiple-Camera Pose Estimation Toward Robotic Applications

Efficient Multi-person Hierarchical 3D Pose Estimation for Autonomous Driving

Robust 3D Reconstruction with an RGB-D Camera

High-precision Human Body Acquisition Via Multi-View Binocular Stereopsis

Visual Odometry Based 3D-Reconstruction

Unsupervised Universal Hierarchical Multi-Person 3D Pose Estimation for Natural Scenes

3D Articulated Skeleton Extraction Using a Single Consumer-Grade Depth Camera.

A Survey on 3D Hand Pose Estimation: Cameras, Methods, and Datasets

A Novel Extrinsic Calibration Method of Mobile Manipulator Camera and 2D-LiDAR via Arbitrary Trihedron-Based Reconstruction

Robust Estimation of 3D Human Poses from a Single Image

Resolving Position Ambiguity of IMU-Based Human Pose with a Single RGB Camera

Kinematics-based 3D Human-Object Interaction Reconstruction from Single View

Robust 3D Human Pose Estimation from Single Images or Video Sequences

RGBManip: Monocular Image-based Robotic Manipulation through Active Object Pose Estimation

Reconstructing 3D human pose and shape from a single image and sparse IMUs

3D Foundation Models Enable Simultaneous Geometry and Pose Estimation of Grasped Objects

A 3D Hand Attitude Estimation Method for Fixed Hand Posture Based on Dual-View RGB Images