Abstract:The spatial resolution of hyperspectral images (HSIs) is usually limited due to internal imaging mechanisms. To obtain imagery with high spectral and high spatial resolutions, which is essential for subsequent HSI processing tasks, a costeffective approach is to fuse HSI with multispectral images (MSIs). One highly effective fusion method is the convolutional neural network (CNN). However, CNNs have limitations in capturing global information and complex features. Recently, visual transformers (ViTs) have garnered interest for their ability to process non-local information. Despite this, existing HSI-MSI fusion methods suffer from insufficient spatial-spectral feature interaction, resulting in suboptimal fusion quality. To address these challenges, we propose a multiscale deep cross-fusion transformer (MDC-FusFormer) network for HSI and MSI fusion. This network effectively performs the interactive fusion of spatial-spectral features, thereby enhancing the quality of the fused images. MDC-FusFormer employs a three-branch network architecture consisting of two independent progressive feature mining modules (PFMMs), a multiscale deep cross-fusion attention module, and a spatial-spectral feature fusion module. Initially, shallow features at different scales of MSI and HSI are recursively extracted through successive up- and down-sampling using CNNs. These features then interact with the deep cross-modal information at corresponding scales through the attention block. Finally, a multidimensional refinement convolution block (MRCB) is applied to refine the feature information, which is then combined with cascaded up-sampling to reconstruct the high-resolution fused image step by step. Experimental results on five datasets indicate that, compared to nine other methods, MDC-FusFormer delivers superior performance.

Transformer-Based Deep Hashing Method for Multi-Scale Feature Fusion

TransHash: Transformer-based Hamming Hashing for Efficient Image Retrieval

Multi-scale Fusion Transformer Based Weakly Supervised Hashing Learning for Instance Retrieval

Deep Hashing Image Retrieval Based on Hybrid Neural Network and Optimized Metric Learning

Deep Internally Connected Transformer Hashing for Image Retrieval

Depth Image Hashing Algorithm Based on Local Global Feature Fusion

Image Retrieval via Balanced and Maximum Variance Deep Hashing.

Multi-feature Fusion-Based Central Similarity Deep Supervised Hashing.

Multi-Granularity Interactive Transformer Hashing for Cross-modal Retrieval

Deep Multiscale Fusion Hashing for Cross-Modal Retrieval

Multiple Hierarchical Deep Hashing for Large Scale Image Retrieval

Multi-Scale Feature Fusion Based on PVTv2 for Deep Hash Remote Sensing Image Retrieval

Deep Hashing Based Fusing Index Method for Large-Scale Image Retrieval

HybridHash: Hybrid Convolutional and Self-Attention Deep Hashing for Image Retrieval

Deep Collaborative Multi-View Hashing for Large-Scale Image Search

Deep Transfer Hashing for Image Retrieval.

Deep Self-Taught Hashing for Image Retrieval

Multi-level Supervised Hashing with Deep Features for Efficient Image Retrieval

Cross-modal retrieval based on multi-dimensional feature fusion hashing

Supervised Deep Hashing for Scalable Face Image Retrieval

MDC-FusFormer: Multiscale Deep Cross-Fusion Transformer Network for Hyperspectral and Multispectral Image Fusion