Abstract:In recent times, notable advancements have been achieved in amalgamating heterogeneous remote sensing imagery to facilitate Earth observation through the adoption of convolutional neural networks. Nonetheless, due to the variety in imaging mechanisms and imbalanced information prevalent among heterogeneous data, the efficacious exploitation of semantic correlation across different modalities for generating discriminative features continues to pose a formidable challenge. Moreover, not all modalities included in the training dataset can be obtained in real-world test scenarios. Hence, the following inquiry arises: How to explicitly leverage semantic correlation between heterogeneous images to construct discriminative features and maintain performance in test scenarios with missing modalities? To address this pressing concern, we propose an innovative assisted learning framework that employs a "teacher-student" architecture equipped with local and global distillation schemes. We partition the framework into two distinct segments, where each segment acquires specific knowledge independently. In terms of local distillation, the teacher network fosters discriminative feature extraction in the student network using a pixel-wise approach, augmented by the inclusion of a regularization factor to ensure the accuracy of knowledge transfer. For global distillation, the student network is motivated to assimilate category-related information derived from the teacher network, thereby further enriching the knowledge encoding. Extensive evaluations of the datasets, utilizing either optical image/Digital Surface Model (DSM) or optical image/Synthetic Aperture Radar (SAR) for land use classification, provide evidence favoring the effectiveness of the proposed method. Code is available at https://github.com/WHUlwb/Assisted_learning .

Multimodal Semantic Collaborative Classification for Hyperspectral Images and LiDAR Data

Remote Sensing Collaborative Classification Using Multimodal Adaptive Modulation Network

Hyperspectral and LiDAR Data Classification Using Spatial Context and De-Redundant Fusion Network.

Multimodal Deep Learning for Semisupervised Classification of Hyperspectral and LiDAR Data

Joint Classification of Hyperspectral and LiDAR Data Based on Position-Channel Cooperative Attention Network

CMSE: Cross-Modal Semantic Enhancement Network for Classification of Hyperspectral and LiDAR Data

Crossmodal Sequential Interaction Network for Hyperspectral and LiDAR Data Joint Classification

Assisted learning for land use classification: The important role of semantic correlation between heterogeneous images

Dual-Branch Dynamic Modulation Network for Hyperspectral and LiDAR Data Classification.

Dual-Branch Feature Fusion Network Based Cross-Modal Enhanced CNN and Transformer for Hyperspectral and LiDAR Classification

Joint Classification of Hyperspectral Images and LiDAR Data Based on Dual-Branch Transformer

Multimodal Remote Sensing Scene Classification Using VLMs and Dual-Cross Attention Networks

Multiview Feature Learning and Multilevel Information Fusion for Joint Classification of Hyperspectral and LiDAR Data

Multiple Information Collaborative Fusion Network for Joint Classification of Hyperspectral and LiDAR Data

A Dual-Branch Deep Learning Architecture for Multisensor and Multitemporal Remote Sensing Semantic Segmentation

Robust Land Cover Classification with Multimodal Knowledge Distillation

SEMANTIC KNOWLEDGE EMBEDDING DEEP LEARNING NETWORK FOR LAND COVER CLASSIFICATION

Land Use Classification Via Multi-Modal Complementary Feature Fusion and Context Information Enhancement for Optical and Sar Images

A Spatial-Spectral Bilinear Representation Fusion Network for Multimodal Classification

Ternary Modality Contrastive Learning for Hyperspectral and LiDAR Data Classification

Collaborative Classification of Hyperspectral and Visible Images with Convolutional Neural Network