Abstract:We consider the problem of estimating frame-level full human body meshes given a video of a person with natural motion dynamics. While much progress in this field has been in single image-based mesh estimation, there has been a recent uptick in efforts to infer mesh dynamics from video given its role in alleviating issues such as depth ambiguity and occlusions. However, a key limitation of existing work is the assumption that all the observed motion dynamics can be modeled using one dynamical/recurrent model. While this may work well in cases with relatively simplistic dynamics, inference with in-the-wild videos presents many challenges. In particular, it is typically the case that different body parts of a person undergo different dynamics in the video, e.g., legs may move in a way that may be dynamically different from hands (e.g., a person dancing). To address these issues, we present a new method for video mesh recovery that divides the human mesh into several local parts following the standard skeletal model. We then model the dynamics of each local part with separate recurrent models, with each model conditioned appropriately based on the known kinematic structure of the human body. This results in a structure-informed local recurrent learning architecture that can be trained in an end-to-end fashion with available annotations. We conduct a variety of experiments on standard video mesh recovery benchmark datasets such as Human3.6M, MPI-INF-3DHP, and 3DPW, demonstrating the efficacy of our design of modeling local dynamics as well as establishing state-of-the-art results based on standard evaluation metrics.

Dynamic Spatio-Temporal Modeling for Example-Based Human Silhouette Recovery

Adaptive Human Silhouette Reconstruction Based on the Exploration of Temporal Information

Temporal Constrained Feasible Subspace Learning for Human Pose Forecasting

LatentHuman: Shape-and-Pose Disentangled Latent Representation for Human Bodies

Dynamic Human Body Reconstruction and Motion Tracking with Low-Cost Depth Cameras

A Spatial-temporal 3D Human Pose Reconstruction Framework

Human Mesh Recovery from Monocular Images via a Skeleton-disentangled Representation

Parametric Human Shape Reconstruction Via Bidirectional Silhouette Guidance

Enhanced Spatio-Temporal Context for Temporally Consistent Robust 3D Human Motion Recovery from Monocular Videos

Topology-Preserved Human Reconstruction with Details

SCAPE-based Human Performance Reconstruction

Self-supervised 3D Human Mesh Recovery from Noisy Point Clouds

Detailed Human Shape Estimation From A Single Image By Hierarchical Mesh Deformation

Dynamic Multi-Person Mesh Recovery From Uncalibrated Multi-View Cameras

Real-time tracking and pose-estimation of walking human from silhouette images

3D Human Pose and Shape Reconstruction from Videos Via Confidence-Aware Temporal Feature Aggregation

Feature-Assisted Dense Spatio-Temporal Reconstruction From Binocular Sequences

ProGraph: Temporally-alignable Probability Guided Graph Topological Modeling for 3D Human Reconstruction

Implicit 3D Human Mesh Recovery using Consistency with Pose and Shape from Unseen-view

STAF: 3D Human Mesh Recovery from Video with Spatio-Temporal Alignment Fusion

Learning Local Recurrent Models for Human Mesh Recovery