Abstract:Land use and land cover maps provide fundamental information that has been used in different types of studies, ranging from public health to carbon cycling. However, the existing remote sensing image classification methods thus far suffer from the insufficient usage of multiple modalities, underconsideration of prior domain knowledge, and poor performance on minority classes. To alleviate these problems, we propose a novel domain knowledge-guided deep collaborative fusion network (DKDFN) with performance boosting for minority categories for land cover classification. More specifically, the DKDFN adopts a multihead encoder and a multibranch decoder structure. The architecture of the encoder probablizes sufficient mining of complementary information from multiple modalities, which are Sentinel-2, Sentinel-1, and SRTM Digital Elevation Data (SRTM) in our case. The multibranch decoder enables land cover classification in a multitask learning setup, performing semantic segmentation and reconstructing multimodal remote sensing indices, which are selected as representatives of domain knowledge. This design incorporates domain knowledge in an effective end-to-end manner. The training stage of our DKDFN is supervised by our proposed asymmetry loss function (ALF), which boosts performance on nearly all categories, especially the categories with a low frequency of occurrence. Ablation studies of the network suggest that our design logic is worth testing in any network with an encoder-decoder structure. The study is conducted in Hunan, China and is verified using a self-labeled multimodal unitemporal remote sensing image dataset. The comparative experiments between DKDFN and 6 state-of-the-art models (U-Net, SegNet, PSPNet, DeepLab, HRNet, MP-ResNet) testify to the superiority of our method and suggest its potential to be applied more widely to map land cover in other geographical areas given the availability of Sentinel-2, Sentinel-1, and SRTM data. The dataset can be downloaded by https://github.com/LauraChow/HunanMultimodalDataset.

Robust Land Cover Classification with Multimodal Knowledge Distillation

Cross-Modal Graph Knowledge Representation and Distillation Learning for Land Cover Classification.

Multimodal Online Knowledge Distillation Framework for Land Use/Cover Classification Using Full or Missing Modalities

Dense Adaptive Grouping Distillation Network for Multimodal Land Cover Classification with Privileged Modality

More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification

Uni-to-Multi Modal Knowledge Distillation for Bidirectional LiDAR-Camera Semantic Segmentation

Cross Modality Knowledge Distillation for Multi-modal Aerial View Object Classification

Multispectral Scene Classification via Cross-Modal Knowledge Distillation

Multisensor Land Cover Classification With Sparsely Annotated Data Based on Convolutional Neural Networks and Self-Distillation

CMR-net: A cross modality reconstruction network for multi-modality remote sensing classification

DKDFN: Domain Knowledge-Guided deep collaborative fusion network for multimodal unitemporal remote sensing land cover classification

Remote Sensing Collaborative Classification Using Multimodal Adaptive Modulation Network

Dual-Branch Feature Fusion Network Based Cross-Modal Enhanced CNN and Transformer for Hyperspectral and LiDAR Classification

Multimodal Deep Learning for Semisupervised Classification of Hyperspectral and LiDAR Data

Hyperspectral and LiDAR Data Classification Using Spatial Context and De-Redundant Fusion Network.

Convolutional Neural Networks for Multimodal Remote Sensing Data Classification

Lightweight High-Resolution Remote Sensing Scene Classification via Adaptive Enhanced Knowledge Distillation.

Joint Classification of Hyperspectral and LiDAR Data Using a Hierarchical CNN and Transformer

Multiple Information Collaborative Fusion Network for Joint Classification of Hyperspectral and LiDAR Data

Multimodal Semantic Collaborative Classification for Hyperspectral Images and LiDAR Data

Diversity-Guided Distillation with Modality-Center Regularization for Robust Multimodal Remote Sensing Image Classification.