Abstract:Multimodal land use/cover classification using optical and synthetic aperture radar (SAR) images has attracted significant attention because the unique radiation and geometric characteristics of these images provide complementary information regarding land properties. However, the significant differences between these modalities create a large semantic gap, posing challenges for effective feature fusion in multimodal learning. Moreover, missing modalities often occur in practical applications due to weather constraints or sensor malfunctions, posing challenges to achieving high performance in cross-modal learning. In this study, we proposed a multimodal online knowledge distillation (MMOKD) framework, designed for land use/cover classification of optical and SAR images using either full or missing modalities. This framework trains one modality-fusion network alongside two modality-specific networks in an end-to-end manner, facilitating both multimodal and cross-modal learning. More specifically, we developed a multimodal feature fusion (MFF) module for integrating heterogeneous features and a single-modal feature generation (SFG) module for encapsulating cross-modal complementary information. In addition, we proposed the joint distillation with multitype fusion knowledge (JD-MFK) method, guiding the modality-specific student networks to comprehensively learn the modality-fusion teacher network. Notably, we adopted an online distillation strategy for real-time feedback and synchronous updates of both modality-fusion and modality-specific networks. Finally, we conducted extensive experiments on two multimodal land use/classification datasets with advanced multimodal fusion, cross-modal distillation, and specific baseline networks for comparison. The results demonstrate the effectiveness of the proposed MMODD, which not only outperforms the other networks in both full- and missing-modality scenarios but also significantly improves model training efficiency.

Chromosome aberrations and the theory of RBE. 3. Evidence from experiments with soft x-rays, and a consideration of the effects of hard x-rays.

Cross-Domain and Cross-Modal Knowledge Distillation in Domain Adaptation for 3D Semantic Segmentation

UniDistill: A Universal Cross-Modality Knowledge Distillation Framework for 3D Object Detection in Bird's-Eye View

CMDFusion: Bidirectional Fusion Network with Cross-modality Knowledge Distillation for LIDAR Semantic Segmentation

Cross-Modality Knowledge Distillation Network for Monocular 3D Object Detection

Unpaired Multi-modal Segmentation via Knowledge Distillation

Robust 3D Semantic Segmentation Method Based on Multi-Modal Collaborative Learning

Multi-to-Single Knowledge Distillation for Point Cloud Semantic Segmentation

Imbalance Knowledge-Driven Multi-modal Network for Land-Cover Semantic Segmentation Using Images and LiDAR Point Clouds

ELiTe: Efficient Image-to-LiDAR Knowledge Transfer for Semantic Segmentation

Knowledge Distillation from 3D to Bird's-Eye-View for LiDAR Semantic Segmentation

Multimodal Online Knowledge Distillation Framework for Land Use/Cover Classification Using Full or Missing Modalities

Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation

Imbalance knowledge-driven multi-modal network for land-cover semantic segmentation using aerial images and LiDAR point clouds

Learnable Cross-modal Knowledge Distillation for Multi-modal Learning with Missing Modality

X$^3$KD: Knowledge Distillation Across Modalities, Tasks and Stages for Multi-Camera 3D Object Detection

A Multi-phase Camera-LiDAR Fusion Network for 3D Semantic Segmentation with Weak Supervision

Joint Semantic Segmentation using representations of LiDAR point clouds and camera images

ProtoTransfer: Cross-Modal Prototype Transfer for Point Cloud Segmentation

Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillation