Abstract:In this paper, we propose a general deep learning based framework, named Sat-MVSF, to perform three-dimensional (3D) reconstruction of the Earth's surface from multi-view optical satellite images. The framework is a complete processing pipeline, including pre-processing, a multi-view stereo (MVS) network for satellite imagery (Sat-MVSNet), and post-processing. The pre-processing handles the geometric and radiometric configuration of the multi-view images and their cropping. The cropped multi-view patches are then fed into Sat-MVSNet, which includes deep feature extraction, rational polynomial camera (RPC) warping, pyramid cost volume construction, regularization, and regression, to obtain the height maps. The error matches are then filtered out and a digital surface model (DSM) is generated in the post-processing. Considering the complexity and diversity of real-world scenes, we also introduce a self-refinement strategy that does not require any ground-truth labels to enhance the performance and robustness of the Sat-MVSF framework. We comprehensively compare the proposed framework with popular commercial software and open-source methods, to demonstrate the potential of the proposed deep learning framework. On the WHU-TLC dataset, where the images are captured with a three-line camera (TLC), the proposed framework outperforms all the other solutions in terms of reconstruction fineness, and also outperforms most of the other methods in terms of efficiency. On the challenging MVS3D dataset, where the images are captured by the WorldView-3 satellite at different times and seasons, the proposed framework also exceeds the existing methods when using the model pretrained on aerial images and the introduced self-refinement strategy, demonstrating a high generalization ability. We also note that the lack of training samples hinders research in this field, and the availability of more high-quality open-source training data will greatly accelerate the research into deep learning based MVS satellite image reconstruction. The code will be available at https://gpcv.whu.edu.cn/data.

Reconstruction of Missing Information in Satellite Images Based on Convolutional Neural Networks

Missing Data Reconstruction in Remote Sensing image with a Unified Spatial-Temporal-Spectral Deep Convolutional Neural Network

Unsupervised missing information reconstruction for single remote sensing image with Deep Code Regression

Spectral Reconstruction From Satellite Multispectral Imagery Using Convolution and Transformer Joint Network

Highly Applicable Iterative Network for 3-D Reconstruction Based on Multiview Satellite Images

A Convolution and Attention Neural Network with MDTW Loss for Cross-Variable Reconstruction of Remote Sensing Image Series

Context-Based Multiscale Unified Network for Missing Data Reconstruction in Remote Sensing Images

For-backward LSTM-based missing data reconstruction for time-series Landsat images

Missing information reconstruction integrating isophote constraint and color-structure control for remote sensing data

Global and Local Dual Fusion Network for Large-Ratio Cloud Occlusion Missing Information Reconstruction of a High-Resolution Remote Sensing Image

Reconstruction of Missing Data in Weather Radar Image Sequences Using Deep Neuron Networks

Aerial Scene Classification Via Multilevel Fusion Based on Deep Convolutional Neural Networks.

Reconstructing Satellites in 3D from Amateur Telescope Images

A Registration-Error-Resistant Swath Reconstruction Method of ZY1-02D Satellite Hyperspectral Data Using SRE-ResNet

Super-resolution Reconstruction of Remote Sensing Images Based on Convolutional Neural Networks.

FVMD-ISRe: 3-D Reconstruction From Few-View Multidate Satellite Images Based on the Implicit Surface Representation of Neural Radiance Fields

Sentinel-2 Cloud Removal Considering Ground Changes by Fusing Multitemporal SAR and Optical Images

Deconstruction of Related Technologies of Ground Image Processing Based on High-Resolution Satellite Remote Sensing Images

Intelligent Reconstruction of Radar Composite Reflectivity Based on Satellite Observations and Deep Learning

A general deep learning based framework for 3D reconstruction from multi-view stereo satellite images

Missing data reconstruction in VHR images based on progressive structure prediction and texture generation