Abstract:Fusion-based hyperspectral image (HSI) super-resolution has become increasingly prevalent for its capability to integrate high-frequency spatial information from the paired high-resolution (HR) RGB reference (Ref-RGB) image. However, most of the existing methods either heavily rely on the accurate alignment between low-resolution (LR) HSIs and RGB images or can only deal with simulated unaligned RGB images generated by rigid geometric transformations, which weakens their effectiveness for real scenes. In this article, we explore the fusion-based HSI super-resolution with real Ref-RGB images that have both rigid and nonrigid misalignments. To properly address the limitations of existing methods for unaligned reference images, we propose an HSI fusion network (HSIFN) with heterogeneous feature extractions, multistage feature alignments, and attentive feature fusion. Specifically, our network first transforms the input HSI and RGB images into two sets of multiscale features with an HSI encoder and an RGB encoder, respectively. The features of Ref-RGB images are then processed by a multistage alignment module to explicitly align the features of Ref-RGB with the LR HSI. Finally, the aligned features of Ref-RGB are further adjusted by an adaptive attention module to focus more on discriminative regions before sending them to the fusion decoder to generate the reconstructed HR HSI. Additionally, we collect a real-world HSI fusion dataset, consisting of paired HSI and unaligned Ref-RGB, to support the evaluation of the proposed model for real scenes. Extensive experiments are conducted on both simulated and our real-world datasets, and it shows that our method obtains a clear improvement over existing single-image and fusion-based super-resolution methods on quantitative assessment as well as visual comparison. The code and dataset are publicly available at https://zeqiang-lai.github.io/HSI-RefSR/.

Arbitrary-Scale Hyperspectral Image Super-Resolution from a Fusion Perspective with Spatial Priors

Multispectral Image Super-Resolution Via RGB Image Fusion and Radiometric Calibration

Hyperspectral Imagery Spatial Super-Resolution Using Generative Adversarial Network.

Remote Sensing Hyperspectral Image Super-Resolution via Multidomain Spatial Information and Multiscale Spectral Information Fusion

Unsupervised Recurrent Hyperspectral Imagery Super-Resolution Using Pixel-Aware Refinement.

Hyperspectral Image Super-Resolution Via Deep Prior Regularization with Parameter Estimation

Hyperspectral Image Super-Resolution with 1D-2D Attentional Convolutional Neural Network.

Universal high spatial resolution hyperspectral imaging using hybrid-resolution image fusion

Deep Spatial-Spectral Information Exploitation For Rapid Hyperspectral Image Super-Resolution

Hyperspectral Image Super-Resolution by Spectral Mixture Analysis and Spatial–Spectral Group Sparsity

Hyperspectral Image Super-Resolution with Optimized RGB Guidance.

A SPATIAL CONSTRAINT AND DEEP LEARNING BASED HYPERSPECTRAL IMAGE SUPER-RESOLUTION METHOD

Hyper-Spectral Image Super-Resolution Using Non-Negative Spectral Representation with Data-Guided Sparsity

Hyperspectral Image Super-Resolution Via Local Low-Rank and Sparse Representations

Dual-Stage Approach Toward Hyperspectral Image Super-Resolution

Hyperspectral Image Super-Resolution via Dual-domain Network Based on Hybrid Convolution

Bayesian Hyperspectral Image Super-Resolution in the Presence of Spectral Variability

Multiscale Factor Joint Learning for Hyperspectral Image Super-Resolution.

Hyperspectral Image Super Resolution With Real Unaligned RGB Guidance

Spatial-Spectral Structured Sparse Low-Rank Representation for Hyperspectral Image Super-Resolution

Multispectral and Hyperspectral Image Fusion with Spatial-Spectral Sparse Representation