Imbalance knowledge-driven multi-modal network for land-cover semantic segmentation using aerial images and LiDAR point clouds

Yameng Wang,Yi Wan,Yongjun Zhang,Bin Zhang,Zhi Gao
DOI: https://doi.org/10.1016/j.isprsjprs.2023.06.014
IF: 12.7
2023-07-07
ISPRS Journal of Photogrammetry and Remote Sensing
Abstract:Despite the good results that have been achieved in unimodal segmentation, the inherent limitations of individual data increase the difficulty of achieving breakthroughs in performance. For that reason, multi-modal learning is increasingly being explored within the field of remote sensing. The present multi-modal methods usually map high-dimensional features to low-dimensional spaces as a preprocess before feature extraction to address the nonnegligible domain gap, which inevitably leads to information loss. To address this issue, in this paper we present our novel I mbalance K nowledge- D riven Multi-modal Net work ( IKD-Net ) to extract features from multi-modal heterogeneous data of aerial images and LiDAR directly. IKD-Net is capable of mining imbalance information across modalities while utilizing a strong modal to drive the feature map refinement of the weaker ones in the global and categorical perspectives by way of two sophisticated plug-and-play modules: the G lobal K nowledge- G uided ( GKG ) and C lass K nowledge- G uided ( CKG ) gated modules. The whole network then is optimized using a joint loss function. While we were developing IKD-Net, we also established a new dataset called the N ational Agriculture Imagery Program and 3 D Elevation Program C ombined dataset in California (N3C-California) , which provides a particular benchmark for multi-modal joint segmentation tasks. In our experiments, IKD-Net outperformed the benchmarks and state-of-the-art methods both in the N3C-California and the small-scale ISPRS Vaihingen dataset. IKD-Net has been ranked first on the real-time leaderboard for the GRSS DFC 2018 challenge evaluation until this paper's submission. Our code and N3C-California dataset are available at https://github.com/wymqqq/IKDNet-pytorch .
imaging science & photographic technology,remote sensing,geography, physical,geosciences, multidisciplinary
What problem does this paper attempt to address?