Abstract:Multi-source remote sensing (RS) image fusion leverages data from various sensors to enhance the accuracy and comprehensiveness of earth observation. Notably, the fusion of hyperspectral (HS) images and light detection and ranging (LiDAR) data has garnered significant attention due to their complementary features. However, current methods predominantly rely on simplistic techniques such as weight sharing, feature superposition, or feature products, which often fall short of achieving true feature fusion. These methods primarily focus on feature accumulation rather than integrative fusion. The Transformer framework, with its self-attention mechanisms, offers potential for effective multimodal data fusion. However, simple linear transformations used in feature extraction may not adequately capture all relevant information. To address these challenges, we propose a novel multimodal contrastive fusion Transformer (MCFT). Our approach employs convolutional neural networks (CNNs) for feature extraction from different modalities and leverages Transformer networks for advanced fusion. We have modified the basic Transformer architecture and propose a double position embedding mode to make it more suitable for RS image processing tasks. We introduce two novel modules: feature alignment module and feature matching module, designed to exploit both paired and unpaired samples. These modules facilitate more effective cross-modal learning by emphasizing the commonalities within the same features and the differences between features from distinct modalities. Experimental evaluations on several publicly available HS-LiDAR datasets demonstrate that proposed method consistently outperforms existing advanced methods. The source code for our approach is available at: https://github.com/SYFYN0317/MCFT.

Modality Fusion Vision Transformer for Hyperspectral and LiDAR Data Collaborative Classification

Classification of hyperspectral and LiDAR data by transformer-based enhancement

Combined Classification of Hyperspectral and LiDAR Data Based on Dual-Channel Cross-Transformer.

MCFT: Multimodal Contrastive Fusion Transformer for Classification of Hyperspectral Image and LiDAR Data

Multimodal Hyperspectral Image Classification via Interconnected Fusion

Mutually Beneficial Transformer for Multimodal Data Fusion

Attention Fusion of Transformer-Based and Scale-Based Method for Hyperspectral and LiDAR Joint Classification

Joint Classification of Hyperspectral Images and LiDAR Data Based on Dual-Branch Transformer

Cross Attention-Based Multi-Scale Convolutional Fusion Network for Hyperspectral and LiDAR Joint Classification

Multiple Information Collaborative Fusion Network for Joint Classification of Hyperspectral and LiDAR Data

A New Multi-Level Attention Feature Fusion Method for Hyperspectral and Lidar Data Joint Classification

LiDAR-Guided Cross-Attention Fusion for Hyperspectral Band Selection and Image Classification

Multimodal Fusion Transformer for Remote Sensing Image Classification

A Joint Convolutional Cross ViT Network for Hyperspectral and Light Detection and Ranging Fusion Classification

Multi-Level Interactive Fusion Network Based on Adversarial Learning for Fusion Classification of Hyperspectral and Lidar Data

HATF: Multi-Modal Feature Learning for Infrared and Visible Image Fusion via Hybrid Attention Transformer

Dual-Branch Feature Fusion Network Based Cross-Modal Enhanced CNN and Transformer for Hyperspectral and LiDAR Classification

Morphological Convolution and Attention Calibration Network for Hyperspectral and LiDAR Data Classification.

A multimodal hyper-fusion transformer for remote sensing image classification

Multimodal Attention-Aware Convolutional Neural Networks for Classification of Hyperspectral and LiDAR Data