Abstract:Surface reconstruction using neural networks has proven effective in reconstructing dense 3D surfaces through image-based neural rendering. Nevertheless, current methods are challenging when dealing with the intricate details of large-scale scenes. The high-fidelity reconstruction performance of neural rendering is constrained by the view sparsity and structural complexity of such scenes. In this paper, we present Res-NeuS, a method combining ResNet-50 and neural surface rendering for dense 3D reconstruction. Specifically, we present appearance embeddings: ResNet-50 is used to extract the appearance depth features of an image to further capture more scene details. We interpolate points near the surface and optimize their weights for the accurate localization of 3D surfaces. We introduce photometric consistency and geometric constraints to optimize 3D surfaces and eliminate geometric ambiguity existing in current methods. Finally, we design a 3D geometry automatic sampling to filter out uninteresting areas and reconstruct complex surface details in a coarse-to-fine manner. Comprehensive experiments demonstrate Res-NeuS's superior capability in the reconstruction of 3D surfaces in complex, large-scale scenes, and the harmful distance of the reconstructed 3D model is 0.4 times that of general neural rendering 3D reconstruction methods and 0.6 times that of traditional 3D reconstruction methods.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to perform high - fidelity surface reconstruction in multi - view complex scenarios. Specifically, existing neural rendering methods face challenges when dealing with large - scale scenes. Especially in cases of sparse views and complex structures, the high - fidelity reconstruction performance is limited. The paper proposes a new method - Res - NeuS, aiming to combine ResNet - 50 and neural implicit surface learning to improve the accuracy and detail - capturing ability of multi - view reconstruction. ### Main Problems 1. **Sparse Views and Structural Complexity**: - Current methods have difficulty capturing fine details when dealing with large - scale scenes due to sparse views and complex structures. - High - fidelity reconstruction performance is limited by sparse views and structural complexity. 2. **Geometric Blur and Color Deviation**: - When only using color information to optimize the scene geometry, there are problems of geometric blur and color deviation. - Traditional volume rendering methods perform poorly when dealing with extreme weather conditions (such as cloudy or foggy days) and remote sensing scene data. ### Solutions The paper proposes the Res - NeuS method, which solves the above problems through the following innovations: 1. **Appearance Embedding**: - Use ResNet - 50 to extract the appearance - depth features of images and further capture more scene details. - Combine these features with view directions and appearance embeddings and input them into the color MLP to enhance the richness of color information. 2. **Interpolation and Weight Optimization**: - Interpolate points near the surface and optimize their weights to achieve accurate positioning of the 3D surface. - Introduce photometric consistency and geometric constraints to optimize the 3D surface and eliminate geometric blur in existing methods. 3. **Automatic Geometric Sampling**: - Design a 3D geometric automatic sampling method to filter out uninteresting areas and gradually reconstruct complex surface details. ### Contributions 1. **Theoretical Analysis of Volume Rendering Deviation**: - Analyze the deviation problems in volume rendering theoretically. 2. **Appearance Embedding Based on Theoretical Analysis**: - Propose an appearance embedding method to optimize the color function. 3. **Combination of Surface Rendering and Volume Rendering**: - Combine surface rendering and volume rendering to make the rendering result nearly unbiased and inversely optimize the SDF network. 4. **Multi - view Stereo Matching Mechanism**: - Integrate a multi - view stereo matching mechanism to constrain the 3D geometric structure. 5. **Novel Geometric Coarse Sampling Strategy**: - Propose a new geometric coarse sampling strategy to efficiently utilize computing resources and view dependencies. Through these innovations, Res - NeuS achieves higher 3D surface reconstruction accuracy in complex large - scale scenes, reduces the geometric blur problem in traditional methods, and simplifies the 3D reconstruction process.

Res-NeuS: Deep Residuals and Neural Implicit Surface Learning for Multi-View Reconstruction

NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction

High-Fidelity Mask-free Neural Surface Reconstruction for Virtual Reality

Recovering Fine Details for Neural Implicit Surface Reconstruction

Dynamic Multi-View Scene Reconstruction Using Neural Implicit Surface

HR-NeuS: Recovering High-Frequency Surface Geometry via Neural Implicit Surfaces

UniRender: Reconstructing 3D Surfaces from Aerial Images with a Unified Rendering Scheme

SC-NeuS: Consistent Neural Surface Reconstruction from Sparse and Noisy Views

NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction

UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction

Geo-Neus: Geometry-Consistent Neural Implicit Surfaces Learning for Multi-view Reconstruction

Fine-detailed Neural Indoor Scene Reconstruction using multi-level importance sampling and multi-view consistency

Edge-aware Neural Implicit Surface Reconstruction

PR-NeuS: A Prior-based Residual Learning Paradigm for Fast Multi-view Neural Surface Reconstruction

Neural 3D Reconstruction in the Wild

NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild

Neuralangelo: High-Fidelity Neural Surface Reconstruction

NeuralRecon: Real-Time Coherent 3D Scene Reconstruction from Monocular Video

GenS: Generalizable Neural Surface Reconstruction from Multi-View Images

Ref-NeuS: Ambiguity-Reduced Neural Implicit Surface Learning for Multi-View Reconstruction with Reflection

Surface-Centric Modeling for High-Fidelity Generalizable Neural Surface Reconstruction