Abstract:3D Semantic-Instance Segmentation (SIS) is a newly emerging research direction that aims to understand visual information of 3D scene on both semantic and instance level. The main difficulty lies in how to coordinate the paradox between mutual aid and sub-optimal problem. Previous methods usually address the mutual aid between instances and semantics by direct feature fusion or hand-crafted constraints to share the common knowledge of the two tasks. However, they neglect the abundant common knowledge of feature context in the feature space. Moreover, the direct feature fusion can raise the sub-optimal problem, since the false prediction of instance object can interfere the prediction of the semantic segmentation and vice versa. To address the above two issues, we propose a novel network of feature context fusion for SIS task, named CF-SIS. The idea is to associatively learn semantic and instance segmentation of 3D point clouds by context fusion with attention in the feature space. Our main contributions are two context fusion modules. First, we propose a novel inter-task context fusion module to take full advantage of mutual aid and relive the sub-optimal problem. It extracts the context in feature space from one task with attention, and selectively fuses the context into the other task using a gate fusion mechanism. Then, in order to enhance the mutual aid effect, the intra-task context fusion module is designed to further integrate the fused context, by selectively merging the similar feature through the self-attention mechanism. We conduct experiments on the S3DIS and ShapeNet datasets and show that CF-SIS outperforms the state-of-the-art methods on semantic and instance segmentation task.

Joint Semantic-Instance Segmentation of 3D Point Clouds: Instance Separation and Semantic Fusion.

Pass3d: Precise And Accelerated Semantic Segmentation For 3d Point Cloud

Associatively Segmenting Instances and Semantics in Point Clouds

JSNet: Joint Instance and Semantic Segmentation of 3D Point Clouds

Associate Semantic-Instance Segmentation of 3D Point Clouds Based on Local Feature Extraction

Joint Point Clouds Semantic and Instance Segmentation by Local Aggregation and Clustering

3DCFS: Fast and Robust Joint 3D Semantic-Instance Segmentation via Coupled Feature Selection

CF-SIS: Semantic-Instance Segmentation of 3D Point Clouds by Context Fusion with Self-Attention

PointMS: Semantic Segmentation for Point Cloud Based on Multi-scale Directional Convolution

JSPNet: Learning Joint Semantic & Instance Segmentation of Point Clouds Via Feature Self-Similarity and Cross-Task Probability

Bi-Directional Attention for Joint Instance and Semantic Segmentation in Point Clouds

3D Object Segmentation Using Cross-Window Point Transformer with Latent Semantic Boundary Guidance

Joint Semantic-Instance Segmentation Method for Intelligent Transportation System

Superpoint-guided Semi-supervised Semantic Segmentation of 3D Point Clouds

Semantic Segmentation-Assisted Instance Feature Fusion for Multi-Level 3D Part Instance Segmentation

Joint Semantic Segmentation using representations of LiDAR point clouds and camera images

Semantic Labeling and Instance Segmentation of 3D Point Clouds Using Patch Context Analysis and Multiscale Processing

PointGS: Bridging and Fusing Geometric and Semantic Space for 3D Point Cloud Analysis

Instance-Aware Embedding for Point Cloud Instance Segmentation

Towards Deeper and Better Multi-view Feature Fusion for 3D Semantic Segmentation

JSMNet Improving Indoor Point Cloud Semantic and Instance Segmentation through Self-Attention and Multiscale