Abstract:Humans can easily deduce the relative pose of an unseen object, without label/training, given only a single query-reference image pair. This is arguably achieved by incorporating (i) 3D/2.5D shape perception from a single image, (ii) render-and-compare simulation, and (iii) rich semantic cue awareness to furnish (coarse) reference-query correspondence. Existing methods implement (i) by a 3D CAD model or well-calibrated multiple images and (ii) by training a network on specific objects, which necessitate laborious ground-truth labeling and tedious training, potentially leading to challenges in generalization. Moreover, (iii) was less exploited in the paradigm of (ii), despite that the coarse correspondence from (iii) enhances the compare process by filtering out non-overlapped parts under substantial pose differences/occlusions. Motivated by this, we propose a novel 3D generalizable relative pose estimation method by elaborating (i) with a 2.5D shape from an RGB-D reference, (ii) with an off-the-shelf differentiable renderer, and (iii) with semantic cues from a pretrained model like DINOv2. Specifically, our differentiable renderer takes the 2.5D rotatable mesh textured by the RGB and the semantic maps (obtained by DINOv2 from the RGB input), then renders new RGB and semantic maps (with back-surface culling) under a novel rotated view. The refinement loss comes from comparing the rendered RGB and semantic maps with the query ones, back-propagating the gradients through the differentiable renderer to refine the 3D relative pose. As a result, our method can be readily applied to unseen objects, given only a single RGB-D reference, without label/training. Extensive experiments on LineMOD, LM-O, and YCB-V show that our training-free method significantly outperforms the SOTA supervised methods, especially under the rigorous Acc@5/10/15° metrics and the challenging cross-dataset settings.

An Extension of Locally Linear Embedding for Pose Estimation of 3D Object

3D Point-to-Keypoint Voting Network for 6D Pose Estimation

Unseen Object Pose Estimation via Registration

Efficient Multi-person Hierarchical 3D Pose Estimation for Autonomous Driving

Learning Stereopsis from Geometric Synthesis for 6D Object Pose Estimation

Zero-Shot 3d Pose Estimation of Unseen Object by Two-Step Rgb-D Fusion

Efficient Encoding and Aligning Viewpoints for 6D Pose Estimation of Unseen Industrial Parts

Robust 3D Human Pose Estimation from Single Images or Video Sequences

Pose-Dependent Low-Rank Embedding for Head Pose Estimation

An Efficient Iterative Pose Estimation Algorithm

Weakly-Supervised Discovery of Geometry-Aware Representation for 3D Human Pose Estimation

Robust Estimation of 3D Human Poses from a Single Image

Pose and Shape Estimation with Discriminatively Learned Parts

LiCamPose: Combining Multi-View LiDAR and RGB Cameras for Robust Single-frame 3D Human Pose Estimation

Leveraging SE(3) Equivariance for Self-Supervised Category-Level Object Pose Estimation

View-Invariant, Occlusion-Robust Probabilistic Embedding for Human Pose

Fast and Robust Multi-Person 3D Pose Estimation from Multiple Views

Towards Human-Level 3D Relative Pose Estimation: Generalizable, Training-Free, with Single Reference

Learning to Search on Manifolds for 3D Pose Estimation of Articulated Objects

Reconstruction and Analysis of Multi-Pose Face Images Based on Nonlinear Dimensionality Reduction

SRPose: Two-view Relative Pose Estimation with Sparse Keypoints