Abstract:Fusion-based hyperspectral image (HSI) super-resolution has become increasingly prevalent for its capability to integrate high-frequency spatial information from the paired high-resolution (HR) RGB reference (Ref-RGB) image. However, most of the existing methods either heavily rely on the accurate alignment between low-resolution (LR) HSIs and RGB images or can only deal with simulated unaligned RGB images generated by rigid geometric transformations, which weakens their effectiveness for real scenes. In this article, we explore the fusion-based HSI super-resolution with real Ref-RGB images that have both rigid and nonrigid misalignments. To properly address the limitations of existing methods for unaligned reference images, we propose an HSI fusion network (HSIFN) with heterogeneous feature extractions, multistage feature alignments, and attentive feature fusion. Specifically, our network first transforms the input HSI and RGB images into two sets of multiscale features with an HSI encoder and an RGB encoder, respectively. The features of Ref-RGB images are then processed by a multistage alignment module to explicitly align the features of Ref-RGB with the LR HSI. Finally, the aligned features of Ref-RGB are further adjusted by an adaptive attention module to focus more on discriminative regions before sending them to the fusion decoder to generate the reconstructed HR HSI. Additionally, we collect a real-world HSI fusion dataset, consisting of paired HSI and unaligned Ref-RGB, to support the evaluation of the proposed model for real scenes. Extensive experiments are conducted on both simulated and our real-world datasets, and it shows that our method obtains a clear improvement over existing single-image and fusion-based super-resolution methods on quantitative assessment as well as visual comparison. The code and dataset are publicly available at https://zeqiang-lai.github.io/HSI-RefSR/.

HyperRefiner: a Refined Hyperspectral Pansharpening Network Based on the Autoencoder and Self-Attention

Hyperspectral and Multispectral Image Fusion Based on Deep Attention Network.

Efficient and Accurate Hyperspectral Pansharpening Using 3D VolumeNet and 2.5D Texture Transfer

A Deep-Shallow Fusion Network With Multidetail Extractor and Spectral Attention for Hyperspectral Pansharpening

Detail Information Prior Net for Remote Sensing Image Pansharpening

Spatial and Spectral Extraction Network With Adaptive Feature Fusion for Pansharpening

ITERATIVE REFINEMENT NETWORK FOR HYPERSPECTRAL IMAGE DENOISING

Hyperspectral Image Super Resolution With Real Unaligned RGB Guidance

HS-McHF: Hypersharpening With Multicomponent-Based Hierarchical Features Fusion Network

Asymmetric Bidirectional Fusion Network for Remote Sensing Pansharpening

A novel cross fusion model with fine-grained detail reconstruction for remote sensing image pan-sharpening

Hyperspectral Image Super-Resolution via Deep Spatiospectral Attention Convolutional Neural Networks

Hyperspectral Image Super-Resolution via Dual-domain Network Based on Hybrid Convolution

Deep Spatial Feedback Refined Network With Multilevel Feature Fusion for Hyperspectral Image Subpixel Mapping

A Triple-Double Convolutional Neural Network for Panchromatic Sharpening

A Multi-Stage Progressive Pansharpening Network Based on Detail Injection with Redundancy Reduction

A self-supervised remote sensing image fusion framework with dual-stage self-learning and spectral super-resolution injection

DCPNet: A Dual-Task Collaborative Promotion Network for Pansharpening

Attention and transformer complementary fusion network for hyperspectral image spectral reconstruction

Hyperspectral Image Super-resolution via Deep Spatio-spectral Convolutional Neural Networks

Attention-Guided Progressive Frequency-Decoupled Network for Pan-Sharpening