Abstract:Estimating 3D human pose and shape from 2D images is a crucial yet challenging task. While prior methods with model-based representations can perform reasonably well on whole-body images, they often fail when parts of the body are occluded or outside the frame. Moreover, these results usually do not faithfully capture the human silhouettes due to their limited representation power of deformable models (e.g., representing only the naked body). An alternative approach is to estimate dense vertices of a predefined template body in the image space. Such representations are effective in localizing vertices within an image but cannot handle out-of-frame body parts. In this work, we learn dense human body estimation that is robust to partial observations. We explicitly model the visibility of human joints and vertices in the x, y, and z axes separately. The visibility in x and y axes help distinguishing out-of-frame cases, and the visibility in depth axis corresponds to occlusions (either self-occlusions or occlusions by other objects). We obtain pseudo ground-truths of visibility labels from dense UV correspondences and train a neural network to predict visibility along with 3D coordinates. We show that visibility can serve as 1) an additional signal to resolve depth ordering ambiguities of self-occluded vertices and 2) a regularization term when fitting a human body model to the predictions. Extensive experiments on multiple 3D human datasets demonstrate that visibility modeling significantly improves the accuracy of human body estimation, especially for partial-body cases. Our project page with code is at: <a class="link-external link-https" href="https://github.com/chhankyao/visdb" rel="external noopener nofollow">this https URL</a>.

Visibility-Aware Human Mesh Recovery Via Balancing Dense Correspondence and Probability Model

MHPro: Multi-hypothesis Probabilistic Modeling for Human Mesh Recovery

VoteHMR: Occlusion-Aware Voting Network for Robust 3D Human Mesh Recovery from Partial Point Clouds

MH‐HMR: Human mesh recovery from monocular images via multi‐hypothesis learning

Human Mesh Recovery from Arbitrary Multi-view Images

CenterHMR: Multi-Person Center-based Human Mesh Recovery

Recovering 3D Human Mesh from Monocular Images: A Survey

Implicit 3D Human Mesh Recovery using Consistency with Pose and Shape from Unseen-view

Learning Visibility for Robust Dense Human Body Estimation

Visibility-guided Human Body Reconstruction from Uncalibrated Multi-view Cameras

Probabilistic Human Mesh Recovery in 3D Scenes from Egocentric Views

Human Mesh Recovery from Monocular Images via a Skeleton-disentangled Representation

InstaHMR: Instance-aware One-stage Multi-person Human Mesh Recovery.

End-to-end Recovery of Human Shape and Pose

GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Cameras

Self-supervised 3D Human Mesh Recovery from Noisy Point Clouds

Simultaneously Recovering Multi-Person Meshes and Multi-View Cameras with Human Semantics

Learning Analytical Posterior Probability for Human Mesh Recovery

Pose2UV: Single-Shot Multiperson Mesh Recovery with Deep UV Prior.

SS-MVMETRO: Semi-supervised multi-view human mesh recovery transformer

Learning Dense UV Completion for Human Mesh Recovery