Abstract:Deep learning has emerged as the predominant approach for multispectral and hyperspectral image fusion. However, most fusion networks are typically trained and validated on hyperspectral and multispectral image pairs generated from the same hyperspectral images, with degradation simulations inconsistent with real situations and relatively limited volumes of images. When transferring a pretrained multispectral and hyperspectral image fusion model from ground or airborne images to spaceborne images, it encounters a larger dataset and more complex spatial-spectral degradation, leading to spectral distortions and spatial artifacts in the fused images. In this article, the challenges associated with the transfer are addressed through the introduction of a self-supervised multispectral and hyperspectral image fusion unrolling network for spaceborne imagery, termed as MH-FUNet. MH-FUNet adopts a self-supervised paradigm to learn a robust mapping from spaceborne data. It utilizes a deep unrolling network to iteratively refine fusion results from coarse to fine. To account for spatial scale differences between the self-supervised training and test datasets, a multiscale fusion strategy is introduced. This strategy is combined with spectral and spatial attention mechanisms to restore spatial and spectral details. Additionally, a gradient constraint unit is proposed to maintain spatial consistency when up-scaling low-resolution hyperspectral imagery. Performance evaluation of the proposed method is conducted against state-of-the-art fusion techniques on both simulated Chikusei dataset and the proposed real WHU-MHF dataset, which consists of simultaneously observed hyperspectral and multispectral image pairs. MH-FUNet outperforms existing methods across all datasets, demonstrating superior performance in spaceborne multispectral and hyperspectral image fusion experiments.

Unsupervised Hyperspectral and Multispectral Image Fusion Based on Deep Adaptive Attention Network

Hyperspectral and Multispectral Image Fusion Based on Deep Attention Network.

A Self-Supervised Spaceborne Multispectral and Hyperspectral Image Fusion Unrolling Network

Unaligned Hyperspectral Image Fusion Via Registration and Interpolation Modeling

Unsupervised Dynamic Convolutional Neural Network Model for Hyperspectral and Multispectral Image Fusion

UMAG-Net: A New Unsupervised Multiattention-Guided Network for Hyperspectral and Multispectral Image Fusion

Hyperspectral and Multispectral Image Fusion Based on Unsupervised Feature Mixing and Reconstruction Network.

Unsupervised Hyperspectral and Multispectral Image Fusion with Deep Spectral-Spatial Collaborative Constraint

SSAU-Net: A Spectral–Spatial Attention-Based U-Net for Hyperspectral Image Fusion

Hyperspectral and Multispectral Image Fusion by Deep Neural Network in a Self-Supervised Manner

Unsupervised Deep Adaptive Learning Spatial Reconstruction Network Based on Hyperspectral Data Fusion

Unsupervised Encoder-Decoder Network Under Spatial and Spectral Guidance for Hyperspectral and Multispectral Image Fusion

Deep Unfolding Network Enhanced by Transformer Priors for Unregistered Hyperspectral and Multispectral Image Fusion

Deep Unsupervised Blind Hyperspectral and Multispectral Data Fusion

A Multiresolution Details Enhanced Attentive Dual-UNet for Hyperspectral and Multispectral Image Fusion

Multispectral and Hyperspectral Image Fusion by MS/HS Fusion Net.

Hyperspectral and Multispectral Image Fusion via Deep Two-Branches Convolutional Neural Network

Hyperspectral and multispectral image fusion via residual selective kernel attention-based U-net

ADMM-HFNet: A Matrix Decomposition-Based Deep Approach for Hyperspectral Image Fusion

Hyperspectral–Multispectral Image Fusion Enhancement Based on Deep Learning

An Unmixing-Based Multi-Attention GAN for Unsupervised Hyperspectral and Multispectral Image Fusion