Abstract:Computer vision has long relied on two kinds of correspondences: pixel correspondences in images and 3D correspondences on object surfaces. Is there another kind, and if there is, what can they do for us? In this paper, we introduce correspondences of the third kind we call reflection correspondences and show that they can help estimate camera pose by just looking at objects without relying on the background. Reflection correspondences are point correspondences in the reflected world, i.e., the scene reflected by the object surface. The object geometry and reflectance alters the scene geometrically and radiometrically, respectively, causing incorrect pixel correspondences. Geometry recovered from each image is also hampered by distortions, namely generalized bas-relief ambiguity, leading to erroneous 3D correspondences. We show that reflection correspondences can resolve the ambiguities arising from these distortions. We introduce a neural correspondence estimator and a RANSAC algorithm that fully leverages all three kinds of correspondences for robust and accurate joint camera pose and object shape estimation just from the object appearance. The method expands the horizon of numerous downstream tasks, including camera pose estimation for appearance modeling (e.g., NeRF) and motion estimation of reflective objects (e.g., cars on the road), to name a few, as it relieves the requirement of overlapping background.

A Correspondence Relaxation Approach for 3D Shape Reconstruction

A synthesis method for personalized 3D face reconstruction

A Simple 3D Reconstruction Method Based on Single Camera

Sketch-based 3-D modeling for piecewise planar objects in single images

Model-driven sketch reconstruction with structure-oriented retrieval

Sparse Representation for 3d Shape Estimation: A Convex Relaxation Approach

3D Shape Estimation from 2D Landmarks: A Convex Relaxation Approach

2D-D Lifting for Shape Reconstruction

Robust 3D Reconstruction with an RGB-D Camera

Convex Relaxation with Log-Determinant Divergence-L1 Regularization for 3D Shape Reconstruction

3D Human Pose and Shape Estimation with Dense Correspondence from a Single Depth Image

A Metric Learning Method for Image-based 3D Shape Retrieval

A Robust 3-D Reconstruction Approach Based on Correspondence Retrieval Using Deep Image Prior

Accurate And Efficient Reconstruction Of 3d Faces From Stereo Images

Robust Geometry and Reflectance Disentanglement for 3D Face Reconstruction from Sparse-view Images

3-D reconstruction using correspondenceless stereo

Disjoint Pose and Shape for 3D Face Reconstruction

Correspondences of the Third Kind: Camera Pose Estimation from Object Reflection

An Image-Based Method for 3D Human Shapes Retrieval.

Monocular 3D Face Reconstruction with Joint 2D and 3D Constraints

Learning 3D Face Reconstruction from a Single Sketch