Abstract:With the rapid advances in autonomous driving, it becomes critical to equip its sensing system with more holistic 3D perception. However, widely explored tasks like 3D detection or point cloud semantic segmentation focus on parsing either the objects (e.g. cars and pedestrians) or scenes (e.g. trees and buildings). In this work, we propose to address the challenging task of LiDAR-based Panoptic Segmentation, which aims to parse both objects and scenes in a unified manner. In particular, we propose Dynamic Shifting Network (DS-Net), which serves as an effective panoptic segmentation framework in the point cloud realm. DS-Net features a dynamic shifting module for complex LiDAR point cloud distributions. We observe that commonly used clustering algorithms like BFS or DBSCAN are incapable of handling complex autonomous driving scenes with non-uniform point cloud distributions and varying instance sizes. Thus, we present an efficient learnable clustering module, dynamic shifting, which adapts kernel functions on the fly for different instances. To further explore the temporal information, we extend the single-scan processing framework to its temporal version, namely 4D-DS-Net, for the task of 4D Panoptic Segmentation, where the same instance across multiple frames should be given the same ID prediction. Instead of naïvely appending a tracking module to DS-Net, we propose to solve the 4D panoptic segmentation in a more unified way. Specifically, 4D-DS-Net first constructs 4D data volume by aligning consecutive LiDAR scans, upon which the temporally unified instance clustering is performed to obtain the final results. Extensive experiments on two large-scale autonomous driving LiDAR datasets, SemanticKITTI and Panoptic nuScenes, are conducted to demonstrate the effectiveness and superior performance of the proposed solution. The code is publicly available at https://github.com/hongfz16/DS-Net.

OccludedInst: an Efficient Instance Segmentation Network for Automatic Driving Occlusion Scenes

CompleteInst: An Efficient Instance Segmentation Network for Missed Detection Scene of Autonomous Driving

Leveraging Front and Side Cues for Occlusion Handling in Monocular 3D Object Detection

Efficient Automatic Driving Instance Segmentation Method Based on Detection

Segmenting Against Dense Distribution and Occlusion in Road Scenarios

A Two-Pipeline Instance Segmentation Network via Boundary Enhancement for Scene Understanding

Occluded Video Instance Segmentation: A Benchmark

OpenInst: A simple query-based method for open-world instance segmentation

PanoSSC: Exploring Monocular Panoptic 3D Scene Reconstruction for Autonomous Driving

Banet: Bidirectional Aggregation Network With Occlusion Handling For Panoptic Segmentation

InstMove: Instance Motion for Object-centric Video Segmentation

LiDAR-based Panoptic Segmentation via Dynamic Shifting Network

AutoInst: Automatic Instance-Based Segmentation of LiDAR 3D Scans

A Study of Occluded Person Re-Identification for Shared Feature Fusion with Pose-Guided and Unsupervised Semantic Segmentation

Spatial Attention Frustum: A 3D Object Detection Method Focusing on Occluded Objects

Visualizing the Invisible: Occluded Vehicle Segmentation and Recovery

LiDAR-based 4D Panoptic Segmentation via Dynamic Shifting Network

Adaptive Query Selection for Camouflaged Instance Segmentation

GHA-Inst: a real-time instance segmentation model utilizing YOLO detection framework

Unified 3D and 4D Panoptic Segmentation via Dynamic Shifting Networks

Adaptive occlusion object detection algorithm based on OL-IoU