Abstract:Multimodal land use/cover classification using optical and synthetic aperture radar (SAR) images has attracted significant attention because the unique radiation and geometric characteristics of these images provide complementary information regarding land properties. However, the significant differences between these modalities create a large semantic gap, posing challenges for effective feature fusion in multimodal learning. Moreover, missing modalities often occur in practical applications due to weather constraints or sensor malfunctions, posing challenges to achieving high performance in cross-modal learning. In this study, we proposed a multimodal online knowledge distillation (MMOKD) framework, designed for land use/cover classification of optical and SAR images using either full or missing modalities. This framework trains one modality-fusion network alongside two modality-specific networks in an end-to-end manner, facilitating both multimodal and cross-modal learning. More specifically, we developed a multimodal feature fusion (MFF) module for integrating heterogeneous features and a single-modal feature generation (SFG) module for encapsulating cross-modal complementary information. In addition, we proposed the joint distillation with multitype fusion knowledge (JD-MFK) method, guiding the modality-specific student networks to comprehensively learn the modality-fusion teacher network. Notably, we adopted an online distillation strategy for real-time feedback and synchronous updates of both modality-fusion and modality-specific networks. Finally, we conducted extensive experiments on two multimodal land use/classification datasets with advanced multimodal fusion, cross-modal distillation, and specific baseline networks for comparison. The results demonstrate the effectiveness of the proposed MMODD, which not only outperforms the other networks in both full- and missing-modality scenarios but also significantly improves model training efficiency.

Diversity-Guided Distillation with Modality-Center Regularization for Robust Multimodal Remote Sensing Image Classification.

Dense Adaptive Grouping Distillation Network for Multimodal Land Cover Classification with Privileged Modality

MSH-Net: Modality-Shared Hallucination With Joint Adaptation Distillation for Remote Sensing Image Classification Using Missing Modalities

Modality-invariant Temporal Representation Learning for Multimodal Sentiment Classification

More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification

Multimodal Online Knowledge Distillation Framework for Land Use/Cover Classification Using Full or Missing Modalities

MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learning

One-stage Modality Distillation for Incomplete Multimodal Learning

Gradient-Guided Modality Decoupling for Missing-Modality Robustness

Multigranularity Decoupling Network With Pseudolabel Selection for Remote Sensing Image Scene Classification

Dynamic-Hierarchical Attention Distillation with Synergetic Instance Selection for Land Cover Classification Using Missing Heterogeneity Images.

Dealing with All-stage Missing Modality: Towards A Universal Model with Robust Reconstruction and Personalization

LDS2AE: Local Diffusion Shared-Specific Autoencoder for Multimodal Remote Sensing Image Classification with Arbitrary Missing Modalities

Robust Multimodal Learning via Representation Decoupling

Diversity Consistency Learning for Remote-Sensing Object Recognition with Limited Labels

Cross-modality Online Distillation for Multi-View Action Recognition

Rethinking Reverse Distillation for Multi-Modal Anomaly Detection

Multimodal Remote Sensing Data Classification Based on Gaussian Mixture Variational Dynamic Fusion Network

Module-wise Adaptive Distillation for Multimodality Foundation Models

Multisource Collaborative Domain Generalization for Cross-Scene Remote Sensing Image Classification

Balanced Multimodal Learning via On-the-fly Gradient Modulation