Abstract:Multi-source remote sensing (RS) image fusion leverages data from various sensors to enhance the accuracy and comprehensiveness of earth observation. Notably, the fusion of hyperspectral (HS) images and light detection and ranging (LiDAR) data has garnered significant attention due to their complementary features. However, current methods predominantly rely on simplistic techniques such as weight sharing, feature superposition, or feature products, which often fall short of achieving true feature fusion. These methods primarily focus on feature accumulation rather than integrative fusion. The Transformer framework, with its self-attention mechanisms, offers potential for effective multimodal data fusion. However, simple linear transformations used in feature extraction may not adequately capture all relevant information. To address these challenges, we propose a novel multimodal contrastive fusion Transformer (MCFT). Our approach employs convolutional neural networks (CNNs) for feature extraction from different modalities and leverages Transformer networks for advanced fusion. We have modified the basic Transformer architecture and propose a double position embedding mode to make it more suitable for RS image processing tasks. We introduce two novel modules: feature alignment module and feature matching module, designed to exploit both paired and unpaired samples. These modules facilitate more effective cross-modal learning by emphasizing the commonalities within the same features and the differences between features from distinct modalities. Experimental evaluations on several publicly available HS-LiDAR datasets demonstrate that proposed method consistently outperforms existing advanced methods. The source code for our approach is available at: https://github.com/SYFYN0317/MCFT.

Research on Multi-Source Heterogeneous Data Fusion Method Based on Cross-Modal Transformer

Research on Multi-Source Heterogeneous Big Data Fusion Method Based on Feature Level

Multi-source heterogeneous data fusion

Feature Fusion Based on Transformer for Cross-modal Retrieval

CrossFormer: Cross-modal Representation Learning via Heterogeneous Graph Transformer

Mutually Beneficial Transformer for Multimodal Data Fusion

A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning

Transformer-Based Multi-Modal Data Fusion Method for COPD Classification and Physiological and Biochemical Indicators Identification

Multi‐modal object detection via transformer network

Combined Classification of Hyperspectral and LiDAR Data Based on Dual-Channel Cross-Transformer.

A Comprehensive Survey on Deep Learning Multi-Modal Fusion: Methods, Technologies and Applications

Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling

A Multi-Source Heterogeneous Data Fusion Method Based on Deep Learning

Deep Symmetric Fusion Transformer for Multimodal Remote Sensing Data Classification

Multi-Feature Fusion Multi-Modal Sentiment Analysis Model Based on Cross-Attention Mechanism

MCFT: Multimodal Contrastive Fusion Transformer for Classification of Hyperspectral Image and LiDAR Data

Classification of hyperspectral and LiDAR data by transformer-based enhancement

MFTransNet: A Multi-Modal Fusion with CNN-Transformer Network for Semantic Segmentation of HSR Remote Sensing Images

TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysis

Deep Multimodal Data Fusion

A General Multi-Source Data Fusion Framework