Abstract:Humans can easily deduce the relative pose of an unseen object, without label/training, given only a single query-reference image pair. This is arguably achieved by incorporating (i) 3D/2.5D shape perception from a single image, (ii) render-and-compare simulation, and (iii) rich semantic cue awareness to furnish (coarse) reference-query correspondence. Existing methods implement (i) by a 3D CAD model or well-calibrated multiple images and (ii) by training a network on specific objects, which necessitate laborious ground-truth labeling and tedious training, potentially leading to challenges in generalization. Moreover, (iii) was less exploited in the paradigm of (ii), despite that the coarse correspondence from (iii) enhances the compare process by filtering out non-overlapped parts under substantial pose differences/occlusions. Motivated by this, we propose a novel 3D generalizable relative pose estimation method by elaborating (i) with a 2.5D shape from an RGB-D reference, (ii) with an off-the-shelf differentiable renderer, and (iii) with semantic cues from a pretrained model like DINOv2. Specifically, our differentiable renderer takes the 2.5D rotatable mesh textured by the RGB and the semantic maps (obtained by DINOv2 from the RGB input), then renders new RGB and semantic maps (with back-surface culling) under a novel rotated view. The refinement loss comes from comparing the rendered RGB and semantic maps with the query ones, back-propagating the gradients through the differentiable renderer to refine the 3D relative pose. As a result, our method can be readily applied to unseen objects, given only a single RGB-D reference, without label/training. Extensive experiments on LineMOD, LM-O, and YCB-V show that our training-free method significantly outperforms the SOTA supervised methods, especially under the rigorous Acc@5/10/15° metrics and the challenging cross-dataset settings.

Methods of Separable RGB-D 3D Human Pose Estimation for Different Scenes

3D Point-to-Keypoint Voting Network for 6D Pose Estimation

Efficient Multi-person Hierarchical 3D Pose Estimation for Autonomous Driving

An RGB-D Based Approach for Human Pose Estimation

Unsupervised Universal Hierarchical Multi-Person 3D Pose Estimation for Natural Scenes

Real-time RGBD-based Extended Body Pose Estimation

Live Stream Temporally Embedded 3D Human Body Pose and Shape Estimation

Motion Capture Research: 3D Human Pose Recovery Based on RGB Video Sequences

Robust 3D Human Pose Estimation from Single Images or Video Sequences

Unified End-to-End YOLOv5-HR-TCM Framework for Automatic 2D/3D Human Pose Estimation for Real-Time Applications

Detecting and tracking people in real time with RGB-D camera

A Survey on Monocular 3D Human Pose Estimation

Relative Pose Estimation for RGB-D Human Input Scans Via Implicit Function Reconstruction

3D Human Pose Estimation = 2D Pose Estimation + Matching

Embodied Scene-aware Human Pose Estimation

Towards Human-Level 3D Relative Pose Estimation: Generalizable, Training-Free, with Single Reference

3D Human Pose Estimation with Single Image and Inertial Measurement Unit (IMU) Sequence

An Effective 3D Human Pose Estimation Method Based on Dilated Convolutions for Videos.

Fast and Robust Multi-Person 3D Pose Estimation from Multiple Views

Multi-Person 3D Pose Estimation from Multi-View Uncalibrated Depth Cameras

Robust Estimation of 3D Human Poses from a Single Image