Learning to Pan-sharpening with Memories of Spatial Details

Maoxun Yuan,Tianyi Zhao,Bo Li,Xingxing Wei

2023-08-08

Abstract:Pan-sharpening, as one of the most commonly used techniques in remote sensing systems, aims to inject spatial details from panchromatic images into multispectral images (MS) to obtain high-resolution multispectral images. Since deep learning has received widespread attention because of its powerful fitting ability and efficient feature extraction, a variety of pan-sharpening methods have been proposed to achieve remarkable performance. However, current pan-sharpening methods usually require the paired panchromatic (PAN) and MS images as input, which limits their usage in some scenarios. To address this issue, in this paper we observe that the spatial details from PAN images are mainly high-frequency cues, i.e., the edges reflect the contour of input PAN images. This motivates us to develop a PAN-agnostic representation to store some base edges, so as to compose the contour for the corresponding PAN image via them. As a result, we can perform the pan-sharpening task with only the MS image when inference. To this end, a memory-based network is adapted to extract and memorize the spatial details during the training phase and is used to replace the process of obtaining spatial information from PAN images when inference, which is called Memory-based Spatial Details Network (MSDN). Finally, we integrate the proposed MSDN module into the existing deep learning-based pan-sharpening methods to achieve an end-to-end pan-sharpening network. With extensive experiments on the Gaofen1 and WorldView-4 satellites, we verify that our method constructs good spatial details without PAN images and achieves the best performance. The code is available at <a class="link-external link-https" href="https://github.com/Zhao-Tian-yi/Learning-to-Pan-sharpening-with-Memories-of-Spatial-Details.git" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Multimedia,Image and Video Processing

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to use only multi - spectral (MS) images for pan - sharpening in the absence of high - resolution panchromatic (PAN) images. Traditional pan - sharpening methods usually require paired PAN and MS images as input, which may be limited in practical applications because the lack of PAN or MS images may be caused by sensor failures and other reasons. Therefore, this paper proposes a new method to extract and remember spatial details through the Memory - based Spatial Details Network (MSDN), and the pan - sharpening task can be completed using only MS images at the test stage, thereby improving data utilization and reducing hardware costs.

Specifically, the main contributions in the paper include:

1. **Proposing a new method for representing spatial details**: This is the first attempt to generate spatial details from a memory network instead of relying on the corresponding PAN image.
2. **Designing the Memory Spatial Details Network (MSDN)**: This network consists of a memory control sub - network and a weighting coefficient sub - network, which are used to generate the required spatial details from MS features. At the same time, a memory - based spatial details learning method is also designed to remember the generation of spatial details during the training process.
3. **Constructing an MSDN - based pan - sharpening framework**: Through extensive experimental verification, this framework has achieved the state - of - the - art qualitative and quantitative performance on Gaofen1 and WorldView - 4 satellite data.

These innovation points of the paper aim to solve the limitations of existing pan - sharpening methods in practical applications, especially to effectively improve the spatial resolution of MS images in the absence of PAN images.