Multi-Modal and Multi-Resolution Data Fusion for High-Resolution Cloud Removal: A Novel Baseline and Benchmark

Fang Xu,Yilei Shi,Patrick Ebel,Wen Yang,Xiao Xiang Zhu

DOI: https://doi.org/10.1109/TGRS.2023.3337845

2024-10-11

Abstract:Cloud removal is a significant and challenging problem in remote sensing, and in recent years, there have been notable advancements in this area. However, two major issues remain hindering the development of cloud removal: the unavailability of high-resolution imagery for existing datasets and the absence of evaluation regarding the semantic meaningfulness of the generated structures. In this paper, we introduce M3R-CR, a benchmark dataset for high-resolution Cloud Removal with Multi-Modal and Multi-Resolution data fusion. With this dataset, we consider the problem of cloud removal in high-resolution optical remote sensing imagery by integrating multi-modal and multi-resolution information. In this context, we have to take into account the alignment errors caused by the multi-resolution nature, along with the more pronounced misalignment issues in high-resolution images due to inherent imaging mechanism differences and other factors. Existing multi-modal data fusion based methods, which assume the image pairs are aligned accurately at pixel-level, are thus not appropriate for this problem. To this end, we design a new baseline named Align-CR to perform the low-resolution SAR image guided high-resolution optical image cloud removal. It gradually warps and fuses the features of the multi-modal and multi-resolution data during the reconstruction process, effectively mitigating concerns associated with misalignment. In the experiments, we evaluate the performance of cloud removal by analyzing the quality of visually pleasing textures using image reconstruction metrics and further analyze the generation of semantically meaningful structures using a well-established semantic segmentation task. The proposed Align-CR method is superior to other baseline methods in both areas.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem this paper attempts to address is the issue of cloud removal in high-resolution remote sensing images. Specifically, the paper points out that current cloud removal techniques face two main challenges:

1. **Lack of high-resolution datasets**: Existing cloud removal datasets are mostly based on medium-resolution Landsat-8 or Sentinel-2 images, which cannot provide the detailed geometric analysis required for high-resolution images, thus limiting the development of high-resolution cloud removal algorithms.
2. **Insufficient evaluation of the semantic significance of generated structures**: Current cloud removal methods mainly rely on visual quality metrics such as PSNR and SSIM to evaluate the quality of reconstructed images, but these metrics cannot comprehensively assess the performance of generated images in subsequent semantic tasks.

To address these issues, the paper proposes a new benchmark dataset called M3R-CR (Multi-Modal and Multi-Resolution Cloud Removal), which includes high-resolution optical observations, radar measurements, and pixel-level land cover annotations sampled globally. Using this dataset, the paper explores how to utilize multi-modal and multi-resolution information for cloud removal in high-resolution optical remote sensing images and proposes a new baseline method called Align-CR to address the issue of inaccurate alignment of multi-resolution data.