TransFusion-CR: Two-Phase SAR-to-Optical Translation and Deep Feature Fusion for Cloud Removal

Rui Liu,Siqi Meng,Yini Peng,Xin Tian

DOI: https://doi.org/10.1109/tgrs.2024.3439854

IF: 8.2

2024-08-21

IEEE Transactions on Geoscience and Remote Sensing

Abstract:Multispectral image (MSI) is essential for remote sensing (RS) tasks. Unfortunately, the presence of clouds will lead to a reduction in the accuracy and availability of ground information. Synthetic aperture radar (SAR) is unaffected by cloud cover, which can serve as a supplemental information source to alleviate the difficulty of cloud removal in MSI. However, SAR images suffer from the problem of speckle noise and contain information only in a single spatial dimension. To achieve cloud removal effectively by fusing MSI and SAR, we propose the TransFusion-CR in this article. It is a novel two-phase SAR and MSI translation-fusion cloud removal network and designed to enhance the information gains provided by SAR. We use a convolutional neural network (CNN) with a large kernel and design special modules to process the global and local information so that TransFusion-CR can reconstruct clear spatial details and accurate spectral information without artifacts. In the translation phase, we establish an SAR frequency extraction (SFE) module based on the discrete wavelet transform (DWT) to separate and process different frequency information of the SAR image. This assists in dealing with the high-frequency speckle noise while generating translated cloud-free MSI with clear spatial information and reliable spectral information. Subsequently, in the fusion phase, a specially designed feature fusion module realizes a weighted triple-source image fusion guided by the cloud mask score (CMS) image, which high-quality cloud-free MSI. Finally, a new loss function is proposed to balance the learning focus and achieve joint optimization. Extensive experiments demonstrate that TransFusion-CR outperforms the state-of-the-art by about 1.3 dB in terms of PSNR when the percentage of thick cloud cover is more than 60% and can construct reliable terrestrial information even in the complete absence of surface data.

imaging science & photographic technology,remote sensing,engineering, electrical & electronic,geochemistry & geophysics