Abstract:In this paper, we propose a novel joint instance and semantic segmentation approach, called JSNet++, to address the instance and semantic segmentation tasks of 3D point clouds simultaneously. We first introduce a basic joint segmentation framework (JSNet). It fuses features from different layers of the backbone network to obtain more discriminative features and makes the two tasks take advantage of each other with a joint instance and semantic segmentation (JISS) module. Specifically, the JISS transforms semantic features into instance embedding space, and then the transformed features are fused with instance features to facilitate instance segmentation. Meanwhile, the JISS module also makes semantic segmentation benefit from instance segmentation by aggregating instance features to semantic feature space. To further reduce the memory consumption of JSNet, we design a dynamic filters for convolution (DFConv) on point clouds. Specifically, we exploit the geometry and density information to generate the dynamic filters, which are used to perform depthwise convolution with the input features. Afterwards, we unify the spatial correlation and channel correlation into a module to fully explore the pointwise correlation in point clouds, and we develop an improved JISS module (JISS*) by using the pointwise correlation module to further improve the accuracy of segmentation. Finally, based on the JSNet, DFConv and JISS*, we propose a new joint segmentation network, termed JSNet++. Experimental results on the benchmarks S3DIS and ScanNet v2 datasets demonstrate the effectiveness of our approach, and our method achieves significant performance improvements over baseline on both instance and semantic segmentation.

Attention-Based Joint Semantic-Instance Segmentation of 3D Point Clouds

JSNet: Joint Instance and Semantic Segmentation of 3D Point Clouds

Bi-Directional Attention for Joint Instance and Semantic Segmentation in Point Clouds

Joint Semantic-Instance Segmentation of 3D Point Clouds: Instance Separation and Semantic Fusion.

Associate Semantic-Instance Segmentation of 3D Point Clouds Based on Local Feature Extraction

Point Attention Network for Semantic Segmentation of 3D Point Clouds

Point Attention Network for Point Cloud Semantic Segmentation.

JSMNet Improving Indoor Point Cloud Semantic and Instance Segmentation through Self-Attention and Multiscale

Superpoint-guided Semi-supervised Semantic Segmentation of 3D Point Clouds

3D Object Segmentation Using Cross-Window Point Transformer with Latent Semantic Boundary Guidance

JSNet++: Dynamic Filters and Pointwise Correlation for 3D Point Cloud Instance and Semantic Segmentation.

3D Reconstruction and Semantic Segmentation Method Combining PointNet and 3D-Lmnet from Single Image

JSENet: Joint Semantic Segmentation and Edge Detection Network for 3D Point Clouds

Attentional Keypoint Detection on Point Clouds for 3D Object Part Segmentation

JSPNet: Learning Joint Semantic & Instance Segmentation of Point Clouds Via Feature Self-Similarity and Cross-Task Probability

Point-based Attention Convolutional Neural Networks for Point Clouds Semantic Segmentation

Semantic Segmentation of Urban Airborne LiDAR Point Clouds Based on Fusion Attention Mechanism and Multi-Scale Features

BEMF-Net: Semantic Segmentation of Large-Scale Point Clouds via Bilateral Neighbor Enhancement and Multi-Scale Fusion

Associatively Segmenting Instances and Semantics in Point Clouds

Densely connected graph convolutional network for joint semantic and instance segmentation of indoor point clouds

Joint Semantic and Instance Segmentation in 3D Point Cloud Based on Transformer