Abstract:Fusion-based hyperspectral image (HSI) super-resolution (SR) aims to recover high-resolution HSI (hr-HSI) from its two degraded modalities, that is, low-resolution HSI (lr-HSI) and high-resolution multispectral image (hr-MSI). The resulting HSI-SR image can be explained from two viewpoints, an inverse problem solution or a spatial–spectral information fusion product. Recent methods focusing on the former point are limited by demands of accurate degradation parameters and precise prior assumptions. On the contrary, the other research line avoids these drawbacks and offers a more flexible design. However, recent methods implicitly handle information fusion. The interaction between lr-HSI and hr-MSI only lies in the extracted feature domain, not directly on the data. Moreover, the proposed modules in recent methods promote information fusion from the perspective of deep learning and neglect the specialty of the HSI domain, leading to weak interpretability and poor reliability in practice. Considering the essence of the HSI-SR problem and the inherent property of HSIs, in this article, we propose a maximum mutual information (MMI) strategy. At first, we model the HSI-SR problem in a new variance inference (VI) architecture. This new VI model simulates the physical process of HSI pairs imaging and provides convenience for the MMI strategy. Then, we insert the MMI strategy in the VI model. The MMI strategy promotes information fusion with quantitative constraints on the information interaction between lr-HSI and hr-MSI. Finally, we implement the VI architecture with a neural network. Benefitting from the MMI strategy, a simple network structure can achieve efficient fusion performance, which indicates that MMI frees DL-based HSI-SR methods from the complicated structure design. The experimental results on synthetic and real datasets demonstrate the superiority of our method to state-of-the-art methods in terms of effectiveness, generalization, and interpretability.

MMIF: Interpretable Hyperspectral and Multispectral Image Fusion Via Maximum Mutual Information

Fusion of infrared and visual images through multiscale hybrid unidirectional total variation

Unaligned Hyperspectral Image Fusion Via Registration and Interpolation Modeling

Fusion of Hyperspectral-Multispectral images joining Spatial-Spectral Dual-Dictionary and structured sparse Low-rank representation

Multispectral and hyperspectral images fusion based on subspace representation and nonlocal low-rank regularization

The Realistic Fusion of Multi-spectral Images

Hyperspectral and Multispectral Image Fusion via Variational Tensor Subspace Decomposition

Implicit Neural Feature Fusion Function for Multispectral and Hyperspectral Image Fusion

Hyperspectral and Multispectral Image Fusion Using the Conditional Denoising Diffusion Probabilistic Model

Unsupervised Blind Spectral-spatial Cross-super-resolution Network for HSI and MSI Fusion

Hyperspectral and multispectral image fusion with arbitrary resolution through self-supervised representations

Recent Advances and New Guidelines on Hyperspectral and Multispectral Image Fusion

Universal high spatial resolution hyperspectral imaging using hybrid-resolution image fusion

Coupled Matrix Factorization Constrained Deep Hyperspectral and Multispectral Image Fusion

Hyperspectral-Multispectral Image Fusion via Tensor Ring and Subspace Decompositions

Multimodal Hyperspectral Image Classification via Interconnected Fusion

Hyperspectral and Multispectral Image Fusion via Nonlocal Low-Rank Tensor Approximation and Sparse Representation

INF3: Implicit Neural Feature Fusion Function for Multispectral and Hyperspectral Image Fusion

A Deep Unfolding Network for Multispectral and Hyperspectral Image Fusion

Semi-blind hyperspectral and multispectral image fusion via generalized inverse matrix optimization

Multi-modal medical image super-resolution fusion based on detail enhancement and weighted local energy deviation