Abstract:Humans can easily deduce the relative pose of an unseen object, without label/training, given only a single query-reference image pair. This is arguably achieved by incorporating (i) 3D/2.5D shape perception from a single image, (ii) render-and-compare simulation, and (iii) rich semantic cue awareness to furnish (coarse) reference-query correspondence. Existing methods implement (i) by a 3D CAD model or well-calibrated multiple images and (ii) by training a network on specific objects, which necessitate laborious ground-truth labeling and tedious training, potentially leading to challenges in generalization. Moreover, (iii) was less exploited in the paradigm of (ii), despite that the coarse correspondence from (iii) enhances the compare process by filtering out non-overlapped parts under substantial pose differences/occlusions. Motivated by this, we propose a novel 3D generalizable relative pose estimation method by elaborating (i) with a 2.5D shape from an RGB-D reference, (ii) with an off-the-shelf differentiable renderer, and (iii) with semantic cues from a pretrained model like DINOv2. Specifically, our differentiable renderer takes the 2.5D rotatable mesh textured by the RGB and the semantic maps (obtained by DINOv2 from the RGB input), then renders new RGB and semantic maps (with back-surface culling) under a novel rotated view. The refinement loss comes from comparing the rendered RGB and semantic maps with the query ones, back-propagating the gradients through the differentiable renderer to refine the 3D relative pose. As a result, our method can be readily applied to unseen objects, given only a single RGB-D reference, without label/training. Extensive experiments on LineMOD, LM-O, and YCB-V show that our training-free method significantly outperforms the SOTA supervised methods, especially under the rigorous Acc@5/10/15° metrics and the challenging cross-dataset settings.

3D Explanation of Photo and Object Recognition Using the Constraint Satisfaction Method

Developing Rigid Constraint for the Estimation of Pose and Structure from a Single Image

Learning Stereopsis from Geometric Synthesis for 6D Object Pose Estimation

Unseen Object Pose Estimation via Registration

Single-View's 3D Explanation Based on Prior Knowledge and Constraint Satisfaction

3D Point-to-Keypoint Voting Network for 6D Pose Estimation

Pose estimation of 3D rigid object based on tentative point correspondence

Iterative pose estimation of a 3D rigid object based on general 2D-3D contour point correspondence

Do not Omit Local Minimizer: a Complete Solution for Pose Estimation from 3D Correspondences

Contour-based Iterative Pose Estimation of 3D Rigid Object

Zero-Shot 3d Pose Estimation of Unseen Object by Two-Step Rgb-D Fusion

An Interactive Image-Based Modeling Approach With Scene Constraints

Complexity of mental geometry for 3D pose perception

A robust direct linear transformation for camera pose estimation using points

Sim2Real Object-Centric Keypoint Detection and Description

Resolving 3D Human Pose Ambiguities with 3D Scene Constraints

An Accurate and Real-time Relative Pose Estimation from Triple Point-line Images by Decoupling Rotation and Translation

Efficient and Outlier-Robust Simultaneous Pose and Correspondence Determination by Branch-and-Bound and Transformation Decomposition

Fast and Robust Multi-Person 3D Pose Estimation from Multiple Views

A Robust and Fast Method to the Perspective-n-Point Problem for Camera Pose Estimation

Towards Human-Level 3D Relative Pose Estimation: Generalizable, Training-Free, with Single Reference