Abstract:Fusing a low spatial resolution hyperspectral image (LR-HIS) with a high spatial resolution multispectral image has become popular for generating a high spatial resolution hyperspectral image (HR-HSI). Most methods assume that the degradation information from high resolution to low resolution is known in spatial and spectral domains. Conversely, this information is often limited or unavailable in practice, restricting their performance. Furthermore, existing fusion methods still face the problem of insufficient exploration of the cross-interaction between the spatial and spectral domains in the HR-HSI, leaving scope for further improvement. This article proposes an unsupervised hybrid network of transformer and CNN (uHNTC) for blind HSI-MSI fusion. The uHNTC comprises three subnetworks: a transformer-based feature fusion subnetwork (FeafusFomer) and two CNN-based degradation subnetworks (SpaDNet and SpeDNet). Considering the strong multilevel spatio-spectral correlation between the desired HR-HSI and the observed images, we design a multilevel cross-feature attention (MCA) mechanism in FeafusFormer. By incorporating the hierarchical spatio-spectral feature fusion into the attention mechanism in the transformer, the MCA globally keeps a high spatio-spectral cross-similarity between the recovered HR-HSI and observed images, thereby ensuring the high cross-interaction of the recovered HR-HSI. Subsequently, the characteristics of degradation information are utilized to guide the design of the SpaDNet and SpeDNet, which helps FeafusFormer accurately recover the desired HR-HSI in complex real-world environments. Through an unsupervised joint training of the three subnetworks, uHNTC recovers the desired HR-HSI without preknown degradation information. Experimental results on three public datasets and a WorldView-2 images show that the uHNTC outperforms ten state-of-the-art fusion methods. Code available: https://github.com/Caoxuheng/HIFtool.

An Efficient Cross-Modality Self-Calibrated Network for Hyperspectral and Multispectral Image Fusion

An Efficient Cross-Modality Self-Calibrated Network for Hyperspectral and Multispectral Image Fusion

Hyperspectral and Multispectral Image Fusion Based on Deep Attention Network.

Coupled Matrix Factorization Constrained Deep Hyperspectral and Multispectral Image Fusion

Hyperspectral and Multispectral Image Fusion via Deep Two-Branches Convolutional Neural Network

Hyperspectral Image Super-Resolution Network Based on Cross-Scale Nonlocal Attention.

Multiscale Spatial Fusion and Regularization Induced Unsupervised Auxiliary Task CNN Model for Deep Super-Resolution of Hyperspectral Images

Unsupervised Hyperspectral and Multispectral Image Fusion with Deep Spectral-Spatial Collaborative Constraint

Cross Attention-Based Multi-Scale Convolutional Fusion Network for Hyperspectral and LiDAR Joint Classification

Fully Dense Multiscale Fusion Network for Hyperspectral Image Classification

A Novel Cross-Scale Octave Network for Hyperspectral and Multispectral Image Fusion

Multiscale Spatial-Spectral Convolutional Network with Image-Based Framework for Hyperspectral Imagery Classification.

Multiscale Feature Fusion Network Incorporating 3D Self-Attention for Hyperspectral Image Classification

Enhanced Multiscale Feature Fusion Network for HSI Classification.

MDC-FusFormer: Multiscale Deep Cross-Fusion Transformer Network for Hyperspectral and Multispectral Image Fusion

Unsupervised Hybrid Network of Transformer and CNN for Blind Hyperspectral and Multispectral Image Fusion

Attention Multihop Graph and Multiscale Convolutional Fusion Network for Hyperspectral Image Classification

Multibranch Feature Fusion Network with Self- and Cross-Guided Attention for Hyperspectral and LiDAR Classification

CNN-Based Multilayer Spatial–Spectral Feature Fusion and Sample Augmentation with Local and Nonlocal Constraints for Hyperspectral Image Classification

Multi-layer feature fusion for hyperspectral image classification