Abstract:Reconstructing accurate 3D scenes from images is a long-standing vision task. Due to the ill-posedness of the single-image reconstruction problem, most well-established methods are built upon multi-view geometry. State-of-the-art (SOTA) monocular metric depth estimation methods can only handle a single camera model and are unable to perform mixed-data training due to the metric ambiguity. Meanwhile, SOTA monocular methods trained on large mixed datasets achieve zero-shot generalization by learning affine-invariant depths, which cannot recover real-world metrics. In this work, we show that the key to a zero-shot single-view metric depth model lies in the combination of large-scale data training and resolving the metric ambiguity from various camera models. We propose a canonical camera space transformation module, which explicitly addresses the ambiguity problems and can be effortlessly plugged into existing monocular models. Equipped with our module, monocular models can be stably trained with over 8 million images with thousands of camera models, resulting in zero-shot generalization to in-the-wild images with unseen camera settings. Experiments demonstrate SOTA performance of our method on 7 zero-shot benchmarks. Notably, our method won the championship in the 2nd Monocular Depth Estimation Challenge. Our method enables the accurate recovery of metric 3D structures on randomly collected internet images, paving the way for plausible single-image metrology. The potential benefits extend to downstream tasks, which can be significantly improved by simply plugging in our model. For example, our model relieves the scale drift issues of monocular-SLAM (Fig. 1), leading to high-quality metric scale dense mapping. The code is available at <a class="link-external link-https" href="https://github.com/YvanYin/Metric3D" rel="external noopener nofollow">this https URL</a>.

Metric depth recovery from monocular images using Shape-from-Shading and specularities

A Non-Lambertian Photometric Stereo under Perspective Projection

Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation

Recovery Of 3d Depth Map From Image Shading For Underwater Applications

A Metric Learning Method for Image-based 3D Shape Retrieval

3-D Shape Recovery of Planar and Curved Surfaces from Shading Cues in Underwater Images

Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image

High-Quality Depth Recovery Via Interactive Multi-view Stereo

Towards Monocular Shape from Refraction

UniDepth: Universal Monocular Metric Depth Estimation

Shading-Based Surface Detail Recovery under General Unknown Illumination

3D reconstruction of non-textured surface by combining shape from shading and stereovision

Recovering Surface Details under General Unknown Illumination Using Shading and Coarse Multi-view Stereo

Shape Recovery of Specular Surface Using Color Highlight Stripe and Light Source Coding

Distilled Visual and Robot Kinematics Embeddings for Metric Depth Estimation in Monocular Scene Reconstruction

A Microfacet-Based Reflectance Model for Photometric Stereo with Highly Specular Surfaces

Single-Shot Metric Depth from Focused Plenoptic Cameras

Proof of Principle for Direct Reconstruction of Qualitative Depth Information from Turbid Media by a Single Hyper Spectral Image

A Sparse-Point Guided Photometric Stereo Method for the Metal Complex Surfaces Measurement and High-Fidelity Geometry Recovery

Dense Depth Recovery Based on Adaptive Image Segmentation

Tackling Challenges of Low-texture and Illumination Variations for Endoscopy Self-supervised Monocular Depth Estimation