Abstract:The current advanced hyperspectral super-resolution methods utilize convolutional neural networks (CNNs) that are either deeper or wider. These networks are designed to acquire end-to-end mapping capability, facilitating the transformation from low-resolution hyperspectral images (LR-HSIs) and high-resolution multispectral images (HR-MSIs) to high-resolution HSIs (HR-HSIs). The existing methods lack the capability to capture details and structures in the image effectively, while multi-input multi-output methods can address this issue efficiently. Therefore, this article proposes a novel network architecture named multi-input multi-output spatial-spectral transformer (MIMO-SST). To apply the multi-input multi-output methods in HSI fusion, specifically integrating the spatial-spectral information of LR-HSI and HR-MSI, we introduce multihead feature map attention, multihead feature channel attention, and a multiscale convolutional gated feedforward network, constructing the proposed mixture spatial-spectral Transformer. Moreover, to enhance the expressive power of image edges and recover the sharpened structure details, this study incorporates a novel wavelet-based high-frequency loss into the ultimate comprehensive loss, with the objective of refining the reconstruction of high-frequency details. Experimental studies on three simulated datasets and one real-world dataset demonstrate that the proposed method in this study outperforms contemporary state-of-the-art methods in terms of performance. It is noteworthy that our method exhibits a 0.85-dB improvement in terms of the peak signal-to-noise ratio (PSNR) metric on the Columbia computer vision laboratory (CAVE) dataset compared to state-of-the-art methods. Our code is publicly available at https://github.com/Freelancefangjian/MIMO-SST.

Multi spectral image fusion by deep convolutional neural network and new spectral loss function

Hyperspectral and Multispectral Image Fusion Based on Deep Attention Network.

Multi-Scale Cross-Attention Fusion Network Based on Image Super-Resolution

An Efficient Cross-Modality Self-Calibrated Network for Hyperspectral and Multispectral Image Fusion

Spatial-Spectral Fusion by Combining Deep Learning and Variation Model

Aerial Scene Classification Via Multilevel Fusion Based on Deep Convolutional Neural Networks.

Hyperspectral and Multispectral Image Fusion Via Self-Supervised Loss and Separable Loss

Hyperspectral and Multispectral Image Fusion by Deep Neural Network in a Self-Supervised Manner

A model-guided deep convolutional sparse coding network for hyperspectral and multispectral image fusion

Multispectral Fusion of RGB and NIR Images Using Weighted Least Squares and Convolution Neural Networks

Deep Convolutional Neural Network for Multi-modal Image Restoration and Fusion

MIMO-SST: Multi-Input Multi-Output Spatial-Spectral Transformer for Hyperspectral and Multispectral Image Fusion

Hyperspectral and Multispectral Image Fusion via Deep Two-Branches Convolutional Neural Network

Going Deeper with Densely Connected Convolutional Neural Networks for Multispectral Pansharpening

Remote Sensing Image Fusion Using Multi-Scale Convolutional Neural Network

A Pseudo-Siamese Deep Convolutional Neural Network for Spatiotemporal Satellite Image Fusion

Multispectral Pan-sharpening via Dual-Channel Convolutional Network with Convolutional LSTM Based Hierarchical Spatial-Spectral Feature Fusion

Multispectral and Hyperspectral Image Fusion Using a 3-D-Convolutional Neural Network

Coupled Matrix Factorization Constrained Deep Hyperspectral and Multispectral Image Fusion

DAFCNN: A Dual-Channel Feature Extraction and Attention Feature Fusion Convolution Neural Network for SAR Image and MS Image Fusion