Transformer Meets GAN: Cloud-Free Multispectral Image Reconstruction via Multisensor Data Fusion in Satellite Images

Congyu Li,Shutao Li,Xinxin Liu

DOI: https://doi.org/10.1109/TGRS.2023.3326545

IF: 8.2

IEEE Transactions on Geoscience and Remote Sensing

Abstract:Cloud-free image reconstruction is of great significance for improving the quality of optical satellite images that are vulnerable to bad weather. When cloud cover makes it impossible to obtain information under the cloud, auxiliary data are indispensable to guide the reconstruction of the cloud-contaminated area. In addition, the areas that require continuous observation are mostly regions with complex features, which puts higher demands on the restoration of texture, color, and other details in data reconstruction. In this article, we propose a transformer-based generative adversarial network for cloud-free multispectral image reconstruction (TransGAN-CFR) via multisensor data fusion in satellite images. Synthetic aperture radar (SAR) images that are not affected by clouds are used as auxiliary data and paired with cloudy optical images into the generative adversarial network (GAN) generator. To take advantage of the deep-shallow features and global–local geographical proximity in remote sensing images, the proposed generator uses a hierarchical encoder–decoder structure, in which the transformer blocks adopt a nonoverlapping window multihead self-attention (WMSA) mechanism and a modified feedforward network (FFN) through depthwise convolutions and the gating mechanism. Besides, we introduce a triplet loss function specifically designed for cloud removal tasks to provide the generated cloud-less image with greater proximity to the ground truth. Compared with seven state-of-the-art deep-learning-based cloud removal models, our network can yield more natural cloud-free images with better visual performance and more accurate results in quantitative evaluation on the SEN12MS-CR dataset.

Computer Science,Engineering,Environmental Science