Abstract:3D scene reconstruction is a long-standing vision task. Existing approaches can be categorized into geometry-based and learning-based methods. The former leverages multi-view geometry but may face catastrophic failures due to the reliance on accurate pixel correspondence across views, while the latter mitigates these issues by learning 2D or 3D representation directly. However, without a large-scale video or 3D training data, it can hardly be generalized to diverse real-world scenarios due to the presence of tens of millions or even billions of optimization parameters in the deep network. Recently, robust monocular depth estimation models trained with large-scale datasets have been proven to possess weak 3D geometry prior, but they are insufficient for reconstruction due to the unknown camera parameters, the affine-invariant property, and inter-frame inconsistency. To address these issues, we propose a novel test-time optimization approach that can transfer the robustness of affine-invariant depth models such as LeReS to challenging diverse scenes while ensuring inter-frame consistency, with only dozens of parameters to optimize per video frame. Specifically, our approach involves freezing the pre-trained affine-invariant depth model's depth predictions, rectifying them by optimizing the unknown scale-shift values with a geometric consistency alignment module, and employing the resulting scale-consistent depth maps to robustly obtain camera poses and achieve dense scene reconstruction, even in low-texture regions. Experiments show that our method achieves state-of-the-art cross-dataset reconstruction on five zero-shot testing datasets. Code is available at: https://aim-uofa.github.io/FrozenRecon/

Zero-Shot Scene Reconstruction from Single Images with Deep Prior Assembly

Joint Learning of Attended Zero-Shot Features and Visual-Semantic Mapping.

Learning 3D Scene Priors with 2D Supervision

ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image

Deep Optimized Priors for 3D Shape Modeling and Reconstruction

Learning Shape Priors for Single-View 3D Completion and Reconstruction

Agent3D-Zero: An Agent for Zero-shot 3D Understanding

PanopticRecon: Leverage Open-vocabulary Instance Segmentation for Zero-shot Panoptic Reconstruction

Zero-Shot Multi-Object Scene Completion

To Boost Zero-Shot Generalization for Embodied Reasoning With Vision-Language Pre-Training

Enhancing Zero-shot 3D Photography Via Mesh-represented Image Inpainting

NopeRoom: Geometric Prior Based Indoor Scene Reconstruction with Unknown Poses

DEEP ZERO-SHOT LEARNING FOR SCENE SKETCH

Zero-Shot Image Harmonization with Generative Model Prior

Zero-Shot Scene Classification for High Spatial Resolution Remote Sensing Images

Semi-supervised Single-view 3D Reconstruction via Multi Shape Prior Fusion Strategy and Self-Attention

ZeroShape: Regression-based Zero-shot Shape Reconstruction

FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models

3D Surface Reconstruction in the Wild by Deforming Shape Priors from Synthetic Data

Delving into Shape-aware Zero-shot Semantic Segmentation

Robust 3D Shape Reconstruction in Zero-Shot from a Single Image in the Wild