Abstract:Generating large-scale and high-quality 3D scene reconstruction from monocular images is an essential technical foundation in augmented reality and robotics. However, the apparent shortcomings (e.g., scale ambiguity, dense depth estimation in texture-less areas) make applying monocular 3D reconstruction to real-world practice challenging. In this work, we combine the advantage of deep learning and multi-view geometry to propose RGB-Fusion, which effectively solves the inherent limitations of traditional monocular reconstruction. To eliminate the confinements of tracking accuracy imposed by the prediction deficiency of neural networks, we propose integrating the PnP (Perspective-n-Point) algorithm into the tracking module. We employ 3D ICP (Iterative Closest Point) matching and 2D feature matching to construct separate error terms and jointly optimize them, reducing the dependence on the accuracy of depth prediction and improving pose estimation accuracy. The approximate pose predicted by the neural network is employed as the initial optimization value to avoid the trapping of local minimums. We formulate a depth map refinement strategy based on the uncertainty of the depth value, which can naturally lead to a refined depth map. Through our method, low-uncertainty elements can significantly update the current depth value while avoiding high-uncertainty elements from adversely affecting depth estimation accuracy. Numerical qualitative and quantitative evaluation results of tracking, depth prediction, and 3D reconstruction show that RGB-Fusion exceeds most monocular 3D reconstruction systems.

InFusionSurf: Refining Neural RGB-D Surface Reconstruction Using Per-Frame Intrinsic Refinement and TSDF Fusion Prior Learning

MD-Surf: Multimodal Neural Surface Reconstruction from Driving Views

SurfelNeRF: Neural Surfel Radiance Fields for Online Photorealistic Reconstruction of Indoor Scenes

GO-Surf: Neural Feature Grid Optimization for Fast, High-Fidelity RGB-D Surface Reconstruction

AniSDF: Fused-Granularity Neural Surfaces with Anisotropic Encoding for High-Fidelity 3D Reconstruction

GSurf: 3D Reconstruction via Signed Distance Fields with Direct Gaussian Supervision

Chunkfusion: A Learning-Based RGB-D 3D Reconstruction Framework Via Chunk-Wise Integration

UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction

HF-NeuS: Improved Surface Reconstruction Using High-Frequency Details

NeuSurf: On-Surface Priors for Neural Surface Reconstruction from Sparse Input Views

NeuralRecon: Real-Time Coherent 3D Scene Reconstruction from Monocular Video

NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstruction

Surface-Centric Modeling for High-Fidelity Generalizable Neural Surface Reconstruction

Real-time 3D Reconstruction System Using Multi-Task Feature Extraction Network and Surfel

RGB-Fusion: Monocular 3D reconstruction with learned depth prediction

RoutedFusion: Learning Real-time Depth Map Fusion

FineRecon: Depth-aware Feed-forward Network for Detailed 3D Reconstruction

Res-NeuS: Deep Residuals and Neural Implicit Surface Learning for Multi-View Reconstruction

Sur2f: A Hybrid Representation for High-Quality and Efficient Surface Reconstruction from Multi-view Images

NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video.

Efficient Surfel Fusion Using Normalised Information Distance