Detail Information Prior Net for Remote Sensing Image Pansharpening

Yuchen Xie,Wei Wu,Haiping Yang,Ning Wu,Ying Shen

DOI: https://doi.org/10.3390/rs13142800

IF: 5

2021-07-16

Remote Sensing

Abstract:Pansharpening, which fuses the panchromatic (PAN) band with multispectral (MS) bands to obtain an MS image with spatial resolution of the PAN images, has been a popular topic in remote sensing applications in recent years. Although the deep-learning-based pansharpening algorithm has achieved better performance than traditional methods, the fusion extracts insufficient spatial information from a PAN image, producing low-quality pansharpened images. To address this problem, this paper proposes a novel progressive PAN-injected fusion method based on superresolution (SR). The network extracts the detail features of a PAN image by using two-stream PAN input; uses a feature fusion unit (FFU) to gradually inject low-frequency PAN features, with high-frequency PAN features added after subpixel convolution; uses a plain autoencoder to inject the extracted PAN features; and applies a structural similarity index measure (SSIM) loss to focus on the structural quality. Experiments performed on different datasets indicate that the proposed method outperforms several state-of-the-art pansharpening methods in both visual appearance and objective indexes, and the SSIM loss can help improve the pansharpened quality on the original dataset.

environmental sciences,imaging science & photographic technology,remote sensing,geosciences, multidisciplinary

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper aims to address the issue of pansharpening in remote sensing image fusion. Specifically, it proposes a novel method to improve the quality of fusion between panchromatic (PAN) and multispectral (MS) images.

#### Main Issues:
1. **Insufficiency of Existing Methods**: Although existing deep learning-based pansharpening algorithms perform better than traditional methods, they fail to fully extract the spatial information of PAN images during the fusion process, resulting in lower quality fused images.
2. **Insufficient Information Extraction**: Traditional pansharpening methods have limited information during multi-scale fusion, leading to some distorted results.

#### Solution:
The paper proposes a new progressive PAN injection fusion method based on super-resolution (SR) (referred to as DIPNet), with the main contributions including:
1. Using dual-stream PAN input to extract PAN features through a convolutional network.
2. Gradually injecting low-frequency PAN features through a Feature Fusion Unit (FFU) and adding high-frequency PAN features after sub-pixel convolution to refine the upsampled MS image.
3. Using a simple autoencoder to inject the extracted PAN features.
4. Using the Structural Similarity Index Measure (SSIM) loss function to guide network training, focusing on the structural quality of the image.

Experiments on multiple datasets show that this method outperforms several existing advanced pansharpening methods in terms of both visual effects and objective metrics, and the SSIM loss helps improve the pansharpening quality on the original datasets.