Abstract:Background Model-based 3D pose estimation has been widely used in many 3D human motion analysis applications, in which vision-based and inertial-based are two distinct lines. Multi-view images in a vision-based markerless capture system provide essential data for motion analysis, but erroneous estimates still occur due to ambiguities, occlusion, or noise in images. Besides, the multi-view setting is hard for the application in the wild. Although inertial measurement units (IMUs) can obtain accurate direction without occlusion, they are usually susceptible to magnetic field interference and drifts. Hybrid motion capture has drawn the attention of researchers in recent years. Existing 3D pose estimation methods jointly optimize the parameters of the 3D pose by minimizing the discrepancy between the image and IMU data. However, these hybrid methods still suffer from the issues such as complex peripheral devices, sensitivity to initialization, and slow convergence. Methods This article presents an approach to improve 3D human pose estimation by fusing a single image with sparse inertial measurement units (IMUs). Based on a dual-stream feature extract network, we design a model-attention network with a residual module to closely couple the dual-modal feature from a static image and sparse inertial measurement units. The final 3D pose and shape parameters are directly obtained by a regression strategy. Results Extensive experiments are conducted on two benchmark datasets for 3D human pose estimation. Compared to state-of-the-art methods, the per vertex error (PVE) of human mesh reduces by 9.4 mm on Total Capture dataset and the mean per joint position error (MPJPE) reduces by 7.8 mm on the Human3.6M dataset. The quantitative comparison demonstrates that the proposed method could effectively fuse sparse IMU data and images and improve pose accuracy.

Face Pose Estimate and Multi-pose Synthesize by 2D Morphable Model

Face Pose Estimation and Synthesis by 2D Morphable Model

Pose Discrimination Based on Cpca-Svm in Dynamic System

Multiple Representations-Based Face Sketch–Photo Synthesis

Realistic 3D Face Modeling by Fusing Multiple 2D Images

Multi-View Face Image Synthesis Using Factorization Model

Face Recognition Based On Pose-Variant Image Synthesis And Multi-Level Multi-Feature Fusion

Pose- and Attribute-consistent Person Image Synthesis

Frontal Face Synthesizing According to Multiple Non-Frontal Inputs and Its Application in Face Recognition.

Face Sketch-Photo Synthesis under Multi-dictionary Sparse Representation Framework

Automatic Face Modeling and Synthesis Based on Image Pairs

Multi-Pose Virtual Try-On Via Self-Adaptive Feature Filtering

Human Multi-View Synthesis from a Single-View Model:Transferred Body and Face Representations

Matching a composite sketch to a photographed face using fused HOG and deep feature models

Self-adaptive Morphable Model Based Multi-View Non-Cooperative 3D Face Reconstruction

Deep pelican based synthesis model for photo-sketch face synthesis and recognition

Multi-Pose Facial Expression Recognition Based on Generative Adversarial Network

PGM-face: Pose-guided Margin Loss for Cross-Pose Face Recognition

Reconstructing 3D human pose and shape from a single image and sparse IMUs

High-Fidelity Face Manipulation With Extreme Poses and Expressions

Recognizing Facial Sketches by Generating Photorealistic Faces Guided by Descriptive Attributes