Abstract:Humans can easily deduce the relative pose of an unseen object, without label/training, given only a single query-reference image pair. This is arguably achieved by incorporating (i) 3D/2.5D shape perception from a single image, (ii) render-and-compare simulation, and (iii) rich semantic cue awareness to furnish (coarse) reference-query correspondence. Existing methods implement (i) by a 3D CAD model or well-calibrated multiple images and (ii) by training a network on specific objects, which necessitate laborious ground-truth labeling and tedious training, potentially leading to challenges in generalization. Moreover, (iii) was less exploited in the paradigm of (ii), despite that the coarse correspondence from (iii) enhances the compare process by filtering out non-overlapped parts under substantial pose differences/occlusions. Motivated by this, we propose a novel 3D generalizable relative pose estimation method by elaborating (i) with a 2.5D shape from an RGB-D reference, (ii) with an off-the-shelf differentiable renderer, and (iii) with semantic cues from a pretrained model like DINOv2. Specifically, our differentiable renderer takes the 2.5D rotatable mesh textured by the RGB and the semantic maps (obtained by DINOv2 from the RGB input), then renders new RGB and semantic maps (with back-surface culling) under a novel rotated view. The refinement loss comes from comparing the rendered RGB and semantic maps with the query ones, back-propagating the gradients through the differentiable renderer to refine the 3D relative pose. As a result, our method can be readily applied to unseen objects, given only a single RGB-D reference, without label/training. Extensive experiments on LineMOD, LM-O, and YCB-V show that our training-free method significantly outperforms the SOTA supervised methods, especially under the rigorous Acc@5/10/15° metrics and the challenging cross-dataset settings.

Perceptual 3D Pose Distance Estimation by Boosting Relational Geometric Features.

Learning a 3D Human Pose Distance Metric from Geometric Pose Descriptor

3D Point-to-Keypoint Voting Network for 6D Pose Estimation

A Geometry-Enhanced 6D Pose Estimation Network with Incomplete Shape Recovery for Industrial Parts

Zero-Shot 3d Pose Estimation of Unseen Object by Two-Step Rgb-D Fusion

PA-Pose: Partial Point Cloud Fusion Based on Reliable Alignment for 6D Pose Tracking

Improving Robustness and Accuracy via Relative Information Encoding in 3D Human Pose Estimation

Lifting by Image -- Leveraging Image Cues for Accurate 3D Human Pose Estimation

PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation

Towards Human-Level 3D Relative Pose Estimation: Generalizable, Training-Free, with Single Reference

A Geometric Knowledge Oriented Single-Frame 2D-to-3D Human Absolute Pose Estimation Method

GS-Pose: Category-Level Object Pose Estimation via Geometric and Semantic Correspondence

6DOF pose estimation of a 3D rigid object based on edge-enhanced point pair features

Feature Boosting Network For 3D Pose Estimation

SD-Pose: facilitating space-decoupled human pose estimation via adaptive pose perception guidance

Towards Precise 3D Human Pose Estimation with Multi-Perspective Spatial-Temporal Relational Transformers

MixedFusion: 6D Object Pose Estimation from Decoupled RGB-Depth Features.

Depth-based Privileged Information for Boosting 3D Human Pose Estimation on RGB

3D WholeBody Pose Estimation based on Semantic Graph Attention Network and Distance Information

Geometric Pose Affordance: 3D Human Pose with Scene Constraints