Abstract:Humans can easily deduce the relative pose of an unseen object, without label/training, given only a single query-reference image pair. This is arguably achieved by incorporating (i) 3D/2.5D shape perception from a single image, (ii) render-and-compare simulation, and (iii) rich semantic cue awareness to furnish (coarse) reference-query correspondence. Existing methods implement (i) by a 3D CAD model or well-calibrated multiple images and (ii) by training a network on specific objects, which necessitate laborious ground-truth labeling and tedious training, potentially leading to challenges in generalization. Moreover, (iii) was less exploited in the paradigm of (ii), despite that the coarse correspondence from (iii) enhances the compare process by filtering out non-overlapped parts under substantial pose differences/occlusions. Motivated by this, we propose a novel 3D generalizable relative pose estimation method by elaborating (i) with a 2.5D shape from an RGB-D reference, (ii) with an off-the-shelf differentiable renderer, and (iii) with semantic cues from a pretrained model like DINOv2. Specifically, our differentiable renderer takes the 2.5D rotatable mesh textured by the RGB and the semantic maps (obtained by DINOv2 from the RGB input), then renders new RGB and semantic maps (with back-surface culling) under a novel rotated view. The refinement loss comes from comparing the rendered RGB and semantic maps with the query ones, back-propagating the gradients through the differentiable renderer to refine the 3D relative pose. As a result, our method can be readily applied to unseen objects, given only a single RGB-D reference, without label/training. Extensive experiments on LineMOD, LM-O, and YCB-V show that our training-free method significantly outperforms the SOTA supervised methods, especially under the rigorous Acc@5/10/15° metrics and the challenging cross-dataset settings.

Monocular Human Pose and Shape Reconstruction Using Part Differentiable Rendering

LatentHuman: Shape-and-Pose Disentangled Latent Representation for Human Bodies

Marker-Less 3d Human Motion Capture With Monocular Image Sequence And Height-Maps

3D Human Reconstruction from A Single Depth Image

Deformable Model-Driven Neural Rendering for High-Fidelity 3D Reconstruction of Human Heads Under Low-View Settings

Learning a Robust Part-Aware Monocular 3D Human Pose Estimator via Neural Architecture Search

Synthetic Training for Monocular Human Mesh Recovery

Towards Human-Level 3D Relative Pose Estimation: Generalizable, Training-Free, with Single Reference

3D human body reconstruction based on SMPL model

Supplementary Materials: Towards Alleviating the Modeling Ambiguity of Unsupervised Monocular 3D Human Pose Estimation

Parametric Human Shape Reconstruction Via Bidirectional Silhouette Guidance

Multi-view Shape Generation for a 3D Human-like Body

Neural Descent for Visual 3D Human Pose and Shape

Weakly Supervised 3D Human Pose and Shape Reconstruction with Normalizing Flows

Human Mesh Recovery from Monocular Images via a Skeleton-disentangled Representation

3D3M: 3D Modulated Morphable Model for Monocular Face Reconstruction

DaNet: Decompose-and-aggregate Network for 3D Human Shape and Pose Estimation

Towards Alleviating the Modeling Ambiguity of Unsupervised Monocular 3D Human Pose Estimation

Shape-Aware Human Pose and Shape Reconstruction Using Multi-View Images

Part123: Part-aware 3D Reconstruction from a Single-view Image

Recovering 3D Human Mesh from Monocular Images: A Survey