Abstract:Joint classification of hyperspectral image (HSI) and light detection and ranging (LiDAR) can simultaneously utilize rich spectral information and elevation information and has become a hot research topic in remote sensing (RS). Although many works have been proposed for this task, their performance cannot reach what we expected due to inadequate cross-modal feature learning and simple feature fusion. This article proposes a multiple information collaborative fusion network (MICF-Net) to overcome those limitations, which aims to leverage the essentially consistent spatial relationships and high-level semantic information in multimodal data to guide the extraction of multimodal fusion features. Specifically, MICF-Net first uses a simple two-branch convolutional neural network (CNN) for preliminary feature extraction. Then, a dual-branch cross-modal attention fusion transformer (CMAFT) is developed to mine global contextual content. By fusing the attention maps of two modalities and limiting their similarity, CMAFT can retain modality-specific information while achieving information interaction based on spatial relationships. Next, an adaptive mask modulation (AMM) module is designed to dynamically balance the learning rate of each modality to ensure the effectiveness of the features of all modalities. Finally, to mine the complementary information of HSI and LiDAR data, a semantic-guided feature fusion (SGFF) module is introduced. It achieves mutual guided learning by exchanging semantic information between two modalities. Positive experimental results counted on three popular HSI and LiDAR datasets demonstrate the effectiveness of the proposed MICF-Net. Our source codes are available at https://github.com/TangXu-Group/Hyperspectral-Images-Classification/tree/main/MICF-Net.

Multi-metric Learning for Multi-Sensor Fusion Based Classification.

Heterogeneous Multi-Metric Learning for Multi-Sensor Fusion.

Multi-view Heterogeneous Fusion and Embedding for Categorical Attributes on Mixed Data.

Multi-model Fusion Metric Learning for Image Set Classification

Multi-metric learning by a pair of twin-metric learning framework

A Novel Multi-Sensor Hybrid Fusion Framework

Kernel Multi-Metric Learning for Multi-Channel Transient Acoustic Signal Classification

Unsupervised metric fusion by cross diffusion

A multiple classifier approach for multisensor data fusion

Multiple classifier systems for multisensor data fusion

Multi-Metrics Classification Machine

A Multi-Sensor Fusion Framework Based on Coupled Residual Convolutional Neural Networks

Multi-Sensor Image Fusion Using Optimized Support Vector Machine and Multiscale Weighted Principal Component Analysis

Multimodal data fusion using signal/image processing methods for multi-class machine learning

Multi-scale Collaborative Coupled Metric Learning Method Based on the Fusion of Class and Structure Information

Multi-view metric learning for visual classification

Multi-Task Sparse Regression Metric Learning For Heterogeneous Classification

A unified multimodal classification framework based on deep metric learning

OPM2L: an Optimal Instance Partition-Based Multi-Metric Learning Method for Heterogeneous Dataset Classification

Multiple Information Collaborative Fusion Network for Joint Classification of Hyperspectral and LiDAR Data

Two-Level Multi-Task Metric Learning with Application to Multi-Classification