Abstract:3D point clouds are rich in geometric structure information, while 2D images contain important and continuous texture information. Combining 2D information to achieve better 3D semantic segmentation has become mainstream in 3D scene understanding. Albeit the success, it still remains elusive how to fuse and process the cross-dimensional features from these two distinct spaces. Existing state-of-the-art usually exploit bidirectional projection methods to align the cross-dimensional features and realize both 2D & 3D semantic segmentation tasks. However, to enable bidirectional mapping, this framework often requires a symmetrical 2D-3D network structure, thus limiting the network's flexibility. Meanwhile, such dual-task settings may distract the network easily and lead to over-fitting in the 3D segmentation task. As limited by the network's inflexibility, fused features can only pass through a decoder network, which affects model performance due to insufficient depth. To alleviate these drawbacks, in this paper, we argue that despite its simplicity, projecting unidirectionally multi-view 2D deep semantic features into the 3D space aligned with 3D deep semantic features could lead to better feature fusion. On the one hand, the unidirectional projection enforces our model focused more on the core task, i.e., 3D segmentation; on the other hand, unlocking the bidirectional to unidirectional projection enables a deeper cross-domain semantic alignment and enjoys the flexibility to fuse better and complicated features from very different spaces. In joint 2D-3D approaches, our proposed method achieves superior performance on the ScanNetv2 benchmark for 3D semantic segmentation.

Multi-view foreground segmentation based on spatial consistency constraints

Object Segmentation Ensuring Consistency Across Multi-Viewpoint Images

Automatic Segmentation of Foreground Objects from Multiple Images Based on Consistency Analysis

Object segmentation based on disparity estimation

3D Object Segmentation Using Cross-Window Point Transformer with Latent Semantic Boundary Guidance

Automatic Object Cosegmentation In Sparse Multiview Images

Simultaneous multi-body stereo and segmentation

Foreground segmentation based on multi-resolution and matting.

Visual Boundary-Guided Pseudo-Labeling for Weakly Supervised 3D Point Cloud Segmentation in Indoor Environments

Efficient video object segmentation based on Gaussian mixture model and Markov random field

Spatio-Temporal Video Segmentation of Static Scenes and Its Applications

Multi-view Object Extraction with Fractional Boundaries

Monocular Video Foreground/Background Segmentation by Tracking Spatial-Color Gaussian Mixture Models

Multi-class Object Segmentation Based on Jointly Integrating Segment-Level and Image-Level Object Priors

Multi-Motion Segmentation: Combining Geometric Model-Fitting and Optical Flow for RGB Sensors

Information Integration for Accurate Foreground Segmentation in Complex Scenes

Joint Multiview Segmentation And Localization Of Rgb-D Images Using Depth-Induced Silhouette Consistency

Multi-guided-based image matting via boundary detection

Virtual Multi-view Fusion for 3D Semantic Segmentation

Video Object Co-Segmentation via Subspace Clustering and Quadratic Pseudo-Boolean Optimization in an MRF Framework

Towards Deeper and Better Multi-view Feature Fusion for 3D Semantic Segmentation