Abstract:The hyperspectral image (HSI) super-resolution (SR) without auxiliary high-resolution images is a challenging task in computer vision applications. The existing methods almost resort to the deep convolutional neural networks of fixed geometrical kernel, which can not model the long-range dependencies and does not conform to the human visual cognition. To address this issue, we propose the cognitively-inspired multi-scale spectral-spatial transformer for HSI SR. To solve the problem of high storage and computation burden, the overlapped band grouping strategy is adopted in light of high similarity between neighboring spectral bands of HSI. Considering the different textures and details that appear in HSIs, inspired by the human cognitive mechanism, the multi-scale spatial and spectral transformer blocks are developed which can efficiently and effectively learn the spatial and spectral feature representation at different scales and long-range dependencies of features. Finally, to fuse the feature information of neighboring groups, the 2D convolution mixed with 3D separable convolution is designed, which fully explores the complementarity and continuity of spatial and spectral information. Extensive experiments conducted on three benchmark datasets demonstrate that the proposed method yields state-of-the-art results at different scales. The effectiveness of the proposed method is verified through spatial and spectral dimension data visualization and ablation experiments. The code and models are publicly available at https://github.com/liushiji666/MMSSTN . The experimental results prove the effectiveness of our proposed method, which largely overcomes the disadvantage that convolution is ineffective for long-range dependence modeling. The method performs long-range dependence modeling on both spatial and spectral features and efficiently mines complementary information between bands, thereby enhancing the model’s high perceptual ability.

MSCSCformer: Multi-scale Convolutional Sparse Coding-based Transformer for Pansharpening

MSCSCformer: Multiscale Convolutional Sparse Coding-Based Transformer for Pansharpening

SSETPAN: Spatial-Spectral Enhanced Transformer Based Network for Pansharpening

DMCSC: Deep Multisource Convolutional Sparse Coding Model for Pansharpening

Multispectral Pan-sharpening via Dual-Channel Convolutional Network with Convolutional LSTM Based Hierarchical Spatial-Spectral Feature Fusion

Transformer-Based Dual-Branch Multiscale Fusion Network for Pan-Sharpening Remote Sensing Images

MDSCNN: Remote Sensing Image Spatial–Spectral Fusion Method via Multi-Scale Dual-Stream Convolutional Neural Network

MSDformer: Multiscale Deformable Transformer for Hyperspectral Image Super-Resolution

Multiscale Spatial-Spectral Interaction Transformer for Pan-Sharpening

PanFormer: a Transformer Based Model for Pan-sharpening

Local-Global Based High-Resolution Spatial-Spectral Representation Network for Pansharpening

STCP: Synergistic Transformer and Convolutional Neural Network for Pansharpening

High-Frequency Transformer Network Based on Window Cross-Attention for Pansharpening.

MSCC-ViT:A Multiscale Visual-Transformer Network Using Convolution Crossing Attention for Hyperspectral Unmixing

A Multiscale and Multidepth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening.

Convolutional LSTM-Based Hierarchical Feature Fusion for Multispectral Pan-Sharpening

CMT: Cross Modulation Transformer with Hybrid Loss for Pansharpening

Going Deeper with Densely Connected Convolutional Neural Networks for Multispectral Pansharpening

A Two-Stream Multiscale Deep Learning Architecture for Pan-Sharpening

Cognitively-Inspired Multi-Scale Spectral-Spatial Transformer for Hyperspectral Image Super-Resolution

Multispectral Pansharpening Based on Multisequence Convolutional Recurrent Neural Network