Abstract:Fusion-based hyperspectral image (HSI) super-resolution has become increasingly prevalent for its capability to integrate high-frequency spatial information from the paired high-resolution (HR) RGB reference (Ref-RGB) image. However, most of the existing methods either heavily rely on the accurate alignment between low-resolution (LR) HSIs and RGB images or can only deal with simulated unaligned RGB images generated by rigid geometric transformations, which weakens their effectiveness for real scenes. In this article, we explore the fusion-based HSI super-resolution with real Ref-RGB images that have both rigid and nonrigid misalignments. To properly address the limitations of existing methods for unaligned reference images, we propose an HSI fusion network (HSIFN) with heterogeneous feature extractions, multistage feature alignments, and attentive feature fusion. Specifically, our network first transforms the input HSI and RGB images into two sets of multiscale features with an HSI encoder and an RGB encoder, respectively. The features of Ref-RGB images are then processed by a multistage alignment module to explicitly align the features of Ref-RGB with the LR HSI. Finally, the aligned features of Ref-RGB are further adjusted by an adaptive attention module to focus more on discriminative regions before sending them to the fusion decoder to generate the reconstructed HR HSI. Additionally, we collect a real-world HSI fusion dataset, consisting of paired HSI and unaligned Ref-RGB, to support the evaluation of the proposed model for real scenes. Extensive experiments are conducted on both simulated and our real-world datasets, and it shows that our method obtains a clear improvement over existing single-image and fusion-based super-resolution methods on quantitative assessment as well as visual comparison. The code and dataset are publicly available at https://zeqiang-lai.github.io/HSI-RefSR/.

Multispectral Image Super-Resolution Via RGB Image Fusion and Radiometric Calibration

Boosting Image Super-Resolution Via Fusion of Complementary Information Captured by Multi-Modal Sensors

Lightweight Multiresolution Feature Fusion Network for Spectral Super-Resolution

Hyperspectral Image Super-Resolution Via Convolutional Neural Network

Hyperspectral Image Super Resolution Reconstruction with a Joint Spectral-Spatial Sub-Pixel Mapping Model.

Reconstructing hyperspectral images from rgb inputs based on intrinsic image decomposition

Remote Sensing Hyperspectral Image Super-Resolution via Multidomain Spatial Information and Multiscale Spectral Information Fusion

When Fusion Meets Super-Resolution: Implicit Edge Calibration for Higher Resolution Multispectral Image Reconstruction

Universal high spatial resolution hyperspectral imaging using hybrid-resolution image fusion

U-Shape Spectral-Transformer for Robust Fusion Based Hyperspectral Super-Resolution

Arbitrary-Scale Hyperspectral Image Super-Resolution from a Fusion Perspective with Spatial Priors

RGB-Induced Feature Modulation Network for Hyperspectral Image Super-Resolution.

Hyperspectral Image Super-Resolution with Optimized RGB Guidance.

Hyperspectral Image Super Resolution With Real Unaligned RGB Guidance

A General Deep Learning Point–Surface Fusion Framework for RGB Image Super-Resolution

MSISR-STF: Spatiotemporal Fusion Via Multilevel Single-Image Super-Resolution

Hyperspectral Image Super-Resolution with a Mosaic RGB Image

Deep Spatial-Spectral Information Exploitation For Rapid Hyperspectral Image Super-Resolution

Multi-sensor Image Super-Resolution with Fuzzy Cluster by Using Multi-Scale and Multi-View Sparse Coding for Infrared Image

A Super-Resolution Algorithm Based on Hybrid Network for Multi-Channel Remote Sensing Images

Super-Resolution Network for Remote Sensing Images Via Preclassification and Deep–Shallow Features Fusion