Abstract:Deep learning has emerged as the predominant approach for multispectral and hyperspectral image fusion. However, most fusion networks are typically trained and validated on hyperspectral and multispectral image pairs generated from the same hyperspectral images, with degradation simulations inconsistent with real situations and relatively limited volumes of images. When transferring a pretrained multispectral and hyperspectral image fusion model from ground or airborne images to spaceborne images, it encounters a larger dataset and more complex spatial-spectral degradation, leading to spectral distortions and spatial artifacts in the fused images. In this article, the challenges associated with the transfer are addressed through the introduction of a self-supervised multispectral and hyperspectral image fusion unrolling network for spaceborne imagery, termed as MH-FUNet. MH-FUNet adopts a self-supervised paradigm to learn a robust mapping from spaceborne data. It utilizes a deep unrolling network to iteratively refine fusion results from coarse to fine. To account for spatial scale differences between the self-supervised training and test datasets, a multiscale fusion strategy is introduced. This strategy is combined with spectral and spatial attention mechanisms to restore spatial and spectral details. Additionally, a gradient constraint unit is proposed to maintain spatial consistency when up-scaling low-resolution hyperspectral imagery. Performance evaluation of the proposed method is conducted against state-of-the-art fusion techniques on both simulated Chikusei dataset and the proposed real WHU-MHF dataset, which consists of simultaneously observed hyperspectral and multispectral image pairs. MH-FUNet outperforms existing methods across all datasets, demonstrating superior performance in spaceborne multispectral and hyperspectral image fusion experiments.

Panchromatic and multi-spectral image fusion for new satellites based on multi-channel deep model

Hyperspectral and Multispectral Image Fusion Based on Deep Attention Network.

A Self-Supervised Spaceborne Multispectral and Hyperspectral Image Fusion Unrolling Network

Study on WorldView-2 Image Fusion Method Based on NMF and HCS Transform

A Hybrid Pan-Sharpening Approach Using Nonnegative Matrix Factorization for WorldView Imageries.

The Study on Image Fusion for Medium and High Spatial Resolution Remote Sensing Images

Fusion Of Multispectral And Panchromatic Images Based On Transferable Parameters

Multispectral and Panchromatic Image Fusion Based on Improved Bilateral Filter

A Multispectral and Panchromatic Images Fusion Method Based on Weighted Mean Curvature Filter Decomposition

Panchromatic and multispectral images fusion based on modified GS-SWT

Study on Panchromatic and Multispectral Image Fusion Based on SFIM and CA Transform

A high-resolution panchromatic-multispectral satellite image fusion method assisted with building segmentation

Fusion of Multispectral and Panchromatic Images Based on a Novel Inter-Band Structure Model

Fusion of Panchromatic and Multispectral Images Using Multiscale Convolution Sparse Decomposition.

Learning Spectral Cues for Multispectral and Panchromatic Image Fusion

Multispectral And Panchromatic Images Fusion Based On Nsct And Gs Transform

Spatial-Spectral Fusion by Combining Deep Learning and Variation Model

FusionCNN: a Remote Sensing Image Fusion Algorithm Based on Deep Convolutional Neural Networks

Unsupervised Hyperspectral and Multispectral Image Fusion with Deep Spectral-Spatial Collaborative Constraint

Integrated Fusion for Panchromatic, Multispectral, Hyperspectral Remote Sensing Images With Different Swath Widths

Spatial–Spectral Fusion by Combining Deep Learning and Variational Model