Abstract:In this paper, we propose a general deep learning based framework, named Sat-MVSF, to perform three-dimensional (3D) reconstruction of the Earth's surface from multi-view optical satellite images. The framework is a complete processing pipeline, including pre-processing, a multi-view stereo (MVS) network for satellite imagery (Sat-MVSNet), and post-processing. The pre-processing handles the geometric and radiometric configuration of the multi-view images and their cropping. The cropped multi-view patches are then fed into Sat-MVSNet, which includes deep feature extraction, rational polynomial camera (RPC) warping, pyramid cost volume construction, regularization, and regression, to obtain the height maps. The error matches are then filtered out and a digital surface model (DSM) is generated in the post-processing. Considering the complexity and diversity of real-world scenes, we also introduce a self-refinement strategy that does not require any ground-truth labels to enhance the performance and robustness of the Sat-MVSF framework. We comprehensively compare the proposed framework with popular commercial software and open-source methods, to demonstrate the potential of the proposed deep learning framework. On the WHU-TLC dataset, where the images are captured with a three-line camera (TLC), the proposed framework outperforms all the other solutions in terms of reconstruction fineness, and also outperforms most of the other methods in terms of efficiency. On the challenging MVS3D dataset, where the images are captured by the WorldView-3 satellite at different times and seasons, the proposed framework also exceeds the existing methods when using the model pretrained on aerial images and the introduced self-refinement strategy, demonstrating a high generalization ability. We also note that the lack of training samples hinders research in this field, and the availability of more high-quality open-source training data will greatly accelerate the research into deep learning based MVS satellite image reconstruction. The code will be available at https://gpcv.whu.edu.cn/data.

SatMVS: A Novel 3D Reconstruction Pipeline for Remote Sensing Satellite Imagery

A general deep learning based framework for 3D reconstruction from multi-view stereo satellite images

Highly Applicable Iterative Network for 3-D Reconstruction Based on Multiview Satellite Images

LODM: Large-scale Online Dense Mapping for UAV

TS-SatMVSNet: Slope Aware Height Estimation for Large-Scale Earth Terrain Multi-view Stereo

Hybrid-MVS: Robust Multi-View Reconstruction with Hybrid Optimization of Visual and Depth Cues

DSM Reconstruction from Uncalibrated Multi-View Satellite Stereo Images by RPC Estimation and Integration

Rational Polynomial Camera Model Warping for Deep Learning Based Satellite Multi-View Stereo Matching

Multi-View Stereo Representation Revist: Region-Aware MVSNet

An Improved TransMVSNet Algorithm for Three-Dimensional Reconstruction in the Unmanned Aerial Vehicle Remote Sensing Domain

SA-SatMVS: Slope Feature-Aware and Across-Scale Information Integration for Large-Scale Earth Terrain Multi-View Stereo

SurfaceNet+: An End-to-end 3D Neural Network for Very Sparse Multi-view Stereopsis

HC-MVSNet: A Probability Sampling-Based Multi-View-stereo Network with Hybrid Cascade Structure for 3D Reconstruction

RS3Mamba: Visual State Space Model for Remote Sensing Images Semantic Segmentation

A Multitask Network for Multiview Stereo Reconstruction: When Semantic Consistency-Based Clustering Meets Depth Estimation Optimization

A Multi-Task Network for Multi-View Stereo Reconstruction: when Semantic Consistency Based Clustering Meets Depth Estimation Optimization

Semantic 3D Reconstruction with Learning MVS and 2D Segmentation of Aerial Images

RS3Mamba: Visual State Space Model for Remote Sensing Image Semantic Segmentation

SDL-MVS: View Space and Depth Deformable Learning Paradigm for Multi-View Stereo Reconstruction in Remote Sensing

Efficient 3D reconstruction for urban scenes

A Novel Recurrent Encoder-Decoder Structure for Large-Scale Multi-View Stereo Reconstruction From an Open Aerial Dataset