Abstract:Spectral super-resolution (SSR) refers to the hyperspectral image (HSI) recovery from an RGB counterpart. Due to the one-to-many nature of the SSR problem, a single RGB image can be reprojected to many HSIs. The key to tackle this ill-posed problem is to plug into multisource prior information such as the natural spatial context prior of RGB images, deep feature prior, or inherent statistical prior of HSIs so as to effectively alleviate the degree of ill-posedness. However, most current approaches only consider the general and limited priors in their customized convolutional neural networks (CNNs), which leads to the inability to guarantee the confidence and fidelity of reconstructed spectra. In this article, we propose a novel holistic prior-embedded relation network (HPRN) to integrate comprehensive priors to regularize and optimize the solution space of SSR. Basically, the core framework is delicately assembled by several multiresidual relation blocks (MRBs) that fully facilitate the transmission and utilization of the low-frequency content prior of RGBs. Innovatively, the semantic prior of RGB inputs is introduced to mark category attributes, and a semantic-driven spatial relation module (SSRM) is invented to perform the feature aggregation of clustered similar ranges for refining recovered characteristics. In addition, we develop a transformer-based channel relation module (TCRM), which breaks the habit of employing scalars as the descriptors of channelwise relations in the previous deep feature prior and replaces them with certain vectors to make the mapping function more robust and smoother. In order to maintain the mathematical correlation and spectral consistency between hyperspectral bands, the second-order prior constraints (SOPCs) are incorporated into the loss function to guide the HSI reconstruction. Finally, extensive experimental results on four benchmarks demonstrate that our HPRN can reach the state-of-the-art performance for SSR quantitatively and qualitatively. Furthermore, the effectiveness and usefulness of the reconstructed spectra are verified by the classification results on the remote sensing dataset. Codes are available at https://github.com/Deep-imagelab/HPRN.

SSR-NET: Spatial–Spectral Reconstruction Network for Hyperspectral and Multispectral Image Fusion

Unsupervised Blind Spectral-spatial Cross-super-resolution Network for HSI and MSI Fusion

Spectral Super-Resolution via Model-Guided Cross-Fusion Network

Multistage Spatial-Spectral Fusion Network for Spectral Super-Resolution

Hyperspectral Image Super-Resolution with Self-Supervised Spectral-Spatial Residual Network

Progressive Spatial Information-Guided Deep Aggregation Convolutional Network for Hyperspectral Spectral Super-Resolution

Universal high spatial resolution hyperspectral imaging using hybrid-resolution image fusion

Unsupervised Hybrid Network of Transformer and CNN for Blind Hyperspectral and Multispectral Image Fusion

An Efficient Cross-Modality Self-Calibrated Network for Hyperspectral and Multispectral Image Fusion

Hyperspectral and Multispectral Image Fusion via Deep Two-Branches Convolutional Neural Network

Fusion of Hyperspectral-Multispectral images joining Spatial-Spectral Dual-Dictionary and structured sparse Low-rank representation

Hyperspectral Image Super-Resolution via Deep Spatiospectral Attention Convolutional Neural Networks

Frequency-Spatial Domain Feature Fusion for Spectral Super-Resolution

Spectral-Spatial Residual Network for Fusing Hyperspectral and Panchromatic Remote Sensing Images

Multispectral and hyperspectral images fusion based on subspace representation and nonlocal low-rank regularization

Spectral–Spatial Feature Extraction Network With SSM–CNN for Hyperspectral–Multispectral Image Collaborative Classification

SLRCNN: Integrating sparse and low-rank with a CNN denoiser for hyperspectral and multispectral image fusion

HMFT: Hyperspectral and Multispectral Image Fusion Super-Resolution Method Based on Efficient Transformer and Spatial-Spectral Attention Mechanism

HPRN: Holistic Prior-Embedded Relation Network for Spectral Super-Resolution

UMTF-Net: An Unsupervised Multiscale Transformer Fusion Network for Hyperspectral and Multispectral Image Fusion