Abstract:Taking into account the limitations of optical imaging, image acquisition equipment is usually designed to make a trade-off between spatial information and spectral information. Hyperspectral image(HSI) can finely identify and classify imaging objects owing to its rich spectral information, while multispectral image(MSI) can provide fine geometric features because of its sufficient spatial information. Hence, fusing HSI and MSI to achieve information complementarity has become a prevalent manner, which increases the reliability and accuracy of the information obtained. However, unlike traditional optical multi-focus image fusion and pan-sharpening of MSI, existing HSI and MSI fusion methods still face with problems in achieving cross-modality information interaction and lack effective utilization of spatial location information. To solve the above problems and to achieve more effective information integration between HSI and MSI, this paper proposes a novel multi-hierarchical cross transformer for hyperspectral and multispectral image fusion (MCT-Net). The proposed MCT-Net consists of two components: (1) a multi-hierarchical cross-modality interacting module (MCIM), which first extracts the deep multi-scale features of HSI and MSI, and then performs cross-modality information interaction between them at identical scales by applying a multi-hierarchical cross transformer (MCT), to reconstruct the spectral information lacking in MSI and the spatial information lacking in HSI; (2) a feature aggregation reconstruction module (FARM) which combines features from MCIM, uses strip convolution to further restore edge features, and reconstructs the fusion results through cascaded upsampling. We conduct comparative experiments on five mainstream HSI datasets to prove the effectiveness and superiority of the proposed method, including the Pavia Center, Pavia University, Urban, Botswana, and Washington DC Mall. For instance, on the Washington DC Mall dataset, compared with the state-of-the-art(SOTA) method in the comparison algorithms, our method improves PSNR by 18.52% and reduces RMSE, ERGAS and SAM by 56.63%, 56.90% and 58.58%, respectively. The source code for MCT-Net can be downloaded from https://github.com/wxy11-27/MCT-Net .

Improving visual grounding with multi-scale discrepancy information and centralized-transformer

ScaleFormer: Revisiting the Transformer-based Backbones from a Scale-wise Perspective for Medical Image Segmentation.

SLViT: Scale-Wise Language-Guided Vision Transformer for Referring Image Segmentation.

Cross-scale Vision Transformer for crowd localization

MAFormer: A transformer network with multi-scale attention fusion for visual recognition

CTMFNet: CNN and Transformer Multiscale Fusion Network of Remote Sensing Urban Scene Imagery

Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Grounding

MCT-Net: Multi-hierarchical cross transformer for hyperspectral and multispectral image fusion

An Efficient and Effective Transformer Decoder-Based Framework for Multi-Task Visual Grounding

Integration Transformer for Ground-Based Cloud Image Segmentation

MFATNet: Multi-Scale Feature Aggregation via Transformer for Remote Sensing Image Change Detection

DTT-CGINet: A Dual Temporal Transformer Network with Multi-Scale Contour-Guided Graph Interaction for Change Detection

Transformer-based Visual Grounding with Cross-modality Interaction

A multimodal hyper-fusion transformer for remote sensing image classification

CTA-Net: A CNN-Transformer Aggregation Network for Improving Multi-Scale Feature Extraction

MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer

Relation Changes Matter: Cross-Temporal Difference Transformer for Change Detection in Remote Sensing Images

MFF-Trans: Multi-level Feature Fusion Transformer for Fine-Grained Visual Classification

IFTSDNet: An Interact-Feature Transformer Network With Spatial Detail Enhancement Module for Change Detection

End-to-end Visual Grounding Based on Query Text Guidance and Multi-stage Reasoning

Remote Sensing Change Detection With Bitemporal and Differential Feature Interactive Perception