Abstract:Deep learning has emerged as the predominant approach for multispectral and hyperspectral image fusion. However, most fusion networks are typically trained and validated on hyperspectral and multispectral image pairs generated from the same hyperspectral images, with degradation simulations inconsistent with real situations and relatively limited volumes of images. When transferring a pretrained multispectral and hyperspectral image fusion model from ground or airborne images to spaceborne images, it encounters a larger dataset and more complex spatial-spectral degradation, leading to spectral distortions and spatial artifacts in the fused images. In this article, the challenges associated with the transfer are addressed through the introduction of a self-supervised multispectral and hyperspectral image fusion unrolling network for spaceborne imagery, termed as MH-FUNet. MH-FUNet adopts a self-supervised paradigm to learn a robust mapping from spaceborne data. It utilizes a deep unrolling network to iteratively refine fusion results from coarse to fine. To account for spatial scale differences between the self-supervised training and test datasets, a multiscale fusion strategy is introduced. This strategy is combined with spectral and spatial attention mechanisms to restore spatial and spectral details. Additionally, a gradient constraint unit is proposed to maintain spatial consistency when up-scaling low-resolution hyperspectral imagery. Performance evaluation of the proposed method is conducted against state-of-the-art fusion techniques on both simulated Chikusei dataset and the proposed real WHU-MHF dataset, which consists of simultaneously observed hyperspectral and multispectral image pairs. MH-FUNet outperforms existing methods across all datasets, demonstrating superior performance in spaceborne multispectral and hyperspectral image fusion experiments.

Multispectral and Hyperspectral Image Fusion Based on Low-Rank Unfolding Network.

A Self-Supervised Spaceborne Multispectral and Hyperspectral Image Fusion Unrolling Network

A Deep Unfolding Network for Multispectral and Hyperspectral Image Fusion

Deep Unfolding Network Enhanced by Transformer Priors for Unregistered Hyperspectral and Multispectral Image Fusion

NMF-DuNet: Nonnegative Matrix Factorization Inspired Deep Unrolling Networks for Hyperspectral and Multispectral Image Fusion

Advancing Hyperspectral and Multispectral Image Fusion: An Information-Aware Transformer-Based Unfolding Network

CCSR-Net: Unfolding Coupled Convolutional Sparse Representation for Multi-focus Image Fusion.

CPP-Net: Embracing Multi-Scale Feature Fusion into Deep Unfolding CP-PPA Network for Compressive Sensing

LR-CSNet: Low-Rank Deep Unfolding Network for Image Compressive Sensing

Spectral–Spatial Dual Graph Unfolding Network for Multispectral and Hyperspectral Image Fusion

FusionNet: An Unsupervised Convolutional Variational Network for Hyperspectral and Multispectral Image Fusion

F-UNet++: Remote Sensing Image Fusion Based on Multipurpose Adaptive Shuffle Attention and Composite Multi-Input Reconstruction Network

Dual-Branch Multiscale Channel Fusion Unfolding Network for Optical Remote Sensing Image Super-Resolution.

Patch-Aware Deep Hyperspectral and Multispectral Image Fusion by Unfolding Subspace-Based Optimization Model

Fusion of Hyperspectral-Multispectral images joining Spatial-Spectral Dual-Dictionary and structured sparse Low-rank representation

Hyperspectral and Multispectral Image Fusion via Nonlocal Low-Rank Tensor Approximation and Sparse Representation

Hyperspectral and multispectral image fusion via residual selective kernel attention-based U-net

ADMM-HFNet: A Matrix Decomposition-Based Deep Approach for Hyperspectral Image Fusion

Unified Cloud-Based Framework for Hyperspectral and Multispectral Image Fusion Incorporating Nonlocal Principles and Tensor Decomposition

A Deep Unfolding Network Based on Intrinsic Image Decomposition for Pansharpening

LRR-Net: An Interpretable Deep Unfolding Network for Hyperspectral Anomaly Detection