Abstract:Spatiotemporal data fusion is a commonly-used and well-proven technique to enhance the application potential of multi-source remote sensing images. However, most existing methods have trouble in generating quality fusion results when areas covered by the images undergoes rapid land cover changes or images have substantial registration errors. While deep learning algorithms have demonstrated their capabilities for imagery fusion, it is challenging to apply deep-learning-based fusion methods in regions that experiences persistent cloud covers and have limited cloud-free imagery observations. To address these challenges, we developed a Multi-scene Spatiotemporal Fusion Network (MUSTFN) algorithm based on a Convolutional Neural Network (CNN). Our approach uses multi-level features to fuse images at different resolutions acquired by multiple sensors. Furthermore, MUSTFN uses the multi-scale features to overcome the effects of geometric registration errors between different images. Additionally, a multi-constrained loss function is proposed to improve the accuracy of imagery fusion over large areas and solve fusion and gap-filling problems simultaneously by utilizing cloud-contaminated images with the fine-tuning method. Compared with several commonly-used methods, our proposed MUSTFN performs better in fusing the 30-m Landsat-7 images and 500-m MODIS images over a small area that has undergone large changes (the average relative Mean Absolute Errors (rMAE) of the first four bands are 6.8% by MUSTFN as compared to 14.1% by the Enhanced Spatial and Temporal Adaptive Reflectance Fusion Model (ESTARFM), 12.8% by the Flexible Spatiotemporal Data Fusion (FSDAF), 8.4% by the Extended Super-Resolution Convolutional Neural Network (ESRCNN), 8.1% by the Spatiotemporal Fusion Using a Generative Adversarial Network (STFGAN)). In particularly for images at different resolutions with different registration accuracies (e.g., 16-m Chinese GaoFen-1 and 500-m MODIS), MUSTFN achieved fusion results of good quality with an average rMAE of 9.3% in spectral reflectance at the first four bands. Finally, we demonstrated the applicability of MUSTFN (average rMAE of 9.18%) when fusing long-term Landsat-8 composite images and MODIS images over a large region (830 km × 600 km). Overall, our results suggest the effectiveness of MUSTFN to address the challenges in imagery fusion, including rapid land cover changes between image acquisition dates, geometric misregistration between images and limited availabilities of cloud-free images. The program of MUSTFN is freely available at: https://github.com/qpyeah/MUSTFN.

A Unified Generative Adversarial Network With Convolution and Transformer for Remote Sensing Image Fusion

Improving Satellite Image Fusion Via Generative Adversarial Training

UFN-GAN: an Unsupervised Generative Adversarial Network for Remote Sensing Image Fusion

Multiresolution Generative Adversarial Networks with Bidirectional Adaptive-Stage Progressive Guided Fusion for Remote Sensing Image

Uncertainty-driven mixture convolution and transformer network for remote sensing image super-resolution

MSFusion: Multistage for Remote Sensing Image Spatiotemporal Fusion Based on Texture Transformer and Convolutional Neural Network

Remote Sensing Image Spatiotemporal Fusion Using a Generative Adversarial Network

Single Remote Sensing Image Super-Resolution Via a Generative Adversarial Network with Stratified Dense Sampling and Chain Training

MLFF-GAN: A Multilevel Feature Fusion With GAN for Spatiotemporal Remote Sensing Images

MDRDA-GAN: A Multi-discriminator Generative Adversarial Network Based on Residual Dense Attention for Remote Sensing Image Fusion

A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation

A multimodal hyper-fusion transformer for remote sensing image classification

DBCT-Net:A Dual Branch Hybrid CNN-transformer Network for Remote Sensing Image Fusion.

Mun-GAN: A Multiscale Unsupervised Network for Remote Sensing Image Pansharpening

Semisupervised Remote Sensing Image Fusion Using Multiscale Conditional Generative Adversarial Network with Siamese Structure.

Deriving High Spatiotemporal Remote Sensing Images Using Deep Convolutional Network

MFFAGAN: Generative Adversarial Network With Multilevel Feature Fusion Attention Mechanism for Remote Sensing Image Super-Resolution

Efficient Adaptive Feature Fusion Network for Remote-Sensing Image Super-Resolution

Generating High-Resolution Hyperspectral Time Series Datasets Based on Unsupervised Spatial-Temporal-spectral Fusion Network Incorporating a Deep Prior

MUSTFN: A Spatiotemporal Fusion Method for Multi-Scale and Multi-Sensor Remote Sensing Images Based on a Convolutional Neural Network

Fusion-UDCGAN: Multifocus Image Fusion via a U-Type Densely Connected Generation Adversarial Network