Cross-modal & Cross-domain Learning for Unsupervised LiDAR Semantic Segmentation

Yiyang Chen,Shanshan Zhao,Changxing Ding,Liyao Tang,Chaoyue Wang,Dacheng Tao

2023-08-05

Abstract:In recent years, cross-modal domain adaptation has been studied on the paired 2D image and 3D LiDAR data to ease the labeling costs for 3D LiDAR semantic segmentation (3DLSS) in the target domain. However, in such a setting the paired 2D and 3D data in the source domain are still collected with additional effort. Since the 2D-3D projections can enable the 3D model to learn semantic information from the 2D counterpart, we ask whether we could further remove the need of source 3D data and only rely on the source 2D images. To answer it, this paper studies a new 3DLSS setting where a 2D dataset (source) with semantic annotations and a paired but unannotated 2D image and 3D LiDAR data (target) are available. To achieve 3DLSS in this scenario, we propose Cross-Modal and Cross-Domain Learning (CoMoDaL). Specifically, our CoMoDaL aims at modeling 1) inter-modal cross-domain distillation between the unpaired source 2D image and target 3D LiDAR data, and 2) the intra-domain cross-modal guidance between the target 2D image and 3D LiDAR data pair. In CoMoDaL, we propose to apply several constraints, such as point-to-pixel and prototype-to-pixel alignments, to associate the semantics in different modalities and domains by constructing mixed samples in two modalities. The experimental results on several datasets show that in the proposed setting, the developed CoMoDaL can achieve segmentation without the supervision of labeled LiDAR data. Ablations are also conducted to provide more analysis. Code will be available publicly.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to address the problem of achieving 3D LiDAR semantic segmentation (3DLSS) in unsupervised cross-modal and cross-domain learning by utilizing only 2D image semantic annotations from the source domain and unannotated 2D images and LiDAR data from the target domain.

#### Main Contributions:
1. **Proposing a New Setting**: The paper investigates a new setting where the source domain contains 2D image data with dense semantic annotations, while the target domain only has unannotated 2D image-LiDAR paired data. To achieve 3DLSS under this setting, a cross-modal and cross-domain learning strategy (CoMoDaL) is proposed to fully exploit the interactions between the two modalities.
2. **Module Design**: CoMoDaL includes two modules: the Inter-Modal Cross-Domain Distillation (ICD) module helps establish correspondences between unpaired 2D source data and 3D target data, enabling more comprehensive 2D-3D interactions; the Inter-Modal Guidance (ICG) module applies point-pixel alignment on enhanced 3D samples to strengthen cross-modal learning within the domain.
3. **Effectiveness Validation**: The method is evaluated on synthetic-to-real adaptation settings (GTA5 to SemanticKITTI) and three real-to-real adaptation settings (A2D2 to SemanticKITTI, Cityscapes to SemanticKITTI, nuScenes Day to Night), demonstrating its effectiveness. Previous related methods are also evaluated under these settings.