Abstract:Training deep models for LiDAR semantic segmentation is challenging due to the inherent sparsity of point clouds. Utilizing temporal data is a natural remedy against the sparsity problem as it makes the input signal denser. However, previous multi-frame fusion algorithms fall short in utilizing sufficient temporal information due to the memory constraint, and they also ignore the informative temporal images. To fully exploit rich information hidden in long-term temporal point clouds and images, we present the Temporal Aggregation Network, termed TASeg. Specifically, we propose a Temporal LiDAR Aggregation and Distillation (TLAD) algorithm, which leverages historical priors to assign different aggregation steps for different classes. It can largely reduce memory and time overhead while achieving higher accuracy. Besides, TLAD trains a teacher injected with gt priors to distill the model, further boosting the performance. To make full use of temporal images, we design a Temporal Image Aggregation and Fusion (TIAF) module, which can greatly expand the camera FOV and enhance the present features. Temporal LiDAR points in the camera FOV are used as mediums to transform temporal image features to the present coordinate for temporal multi-modal fusion. Moreover, we develop a Static-Moving Switch Augmentation (SMSA) algorithm, which utilizes sufficient temporal information to enable objects to switch their motion states freely, thus greatly increasing static and moving training samples. Our TASeg ranks 1st on three challenging tracks, i.e., SemanticKITTI single-scan track, multi-scan track and nuScenes LiDAR segmentation track, strongly demonstrating the superiority of our method. Codes are available at <a class="link-external link-https" href="https://github.com/LittlePey/TASeg" rel="external noopener nofollow">this https URL</a>.

Temporal Feature Matching and Propagation for Semantic Segmentation on 3D Point Cloud Sequences

Learning Temporal Variations for 4D Point Cloud Segmentation

Learning Spatial and Temporal Variations for 4D Point Cloud Segmentation

Pass3d: Precise And Accelerated Semantic Segmentation For 3d Point Cloud

Associate Semantic-Instance Segmentation of 3D Point Clouds Based on Local Feature Extraction

3D Object Segmentation Using Cross-Window Point Transformer with Latent Semantic Boundary Guidance

4D-Former: Multimodal 4D Panoptic Segmentation

PointMS: Semantic Segmentation for Point Cloud Based on Multi-scale Directional Convolution

3D Semantic Segmentation Using Deep Learning for Large-Scale Indoor Point Cloud

TASeg: Temporal Aggregation Network for LiDAR Semantic Segmentation

Human Segmentation with Dynamic LiDAR Data

LiDAR-Based Real-Time Panoptic Segmentation via Spatiotemporal Sequential Data Fusion

MemorySeg: Online LiDAR Semantic Segmentation with a Latent Memory

Dual-Branch Dynamic Object Segmentation Network Based on Spatio-Temporal Information Fusion

A Spatiotemporal Correspondence Approach to Unsupervised LiDAR Segmentation with Traffic Applications

Semantic Segmentation of Point Cloud Scene via Multi-Scale Feature Aggregation and Adaptive Fusion

A Transformer-based Real-time LiDAR Semantic Segmentation Method for Restricted Mobile Devices

SVQNet: Sparse Voxel-Adjacent Query Network for 4D Spatio-Temporal LiDAR Semantic Segmentation

Affinity3D: Propagating Instance-Level Semantic Affinity for Zero-Shot Point Cloud Semantic Segmentation

SegNet4D: Efficient Instance-Aware 4D Semantic Segmentation for LiDAR Point Cloud

Temporal Point Cloud Fusion With Scene Flow for Robust 3D Object Tracking