Abstract:Plane instance segmentation from RGB-D data is a crucial research topic for many downstream tasks. However, most existing deep-learning-based methods utilize only information within the RGB bands, neglecting the important role of the depth band in plane instance segmentation. Based on EfficientSAM, a fast version of SAM, we propose a plane instance segmentation network called PlaneSAM, which can fully integrate the information of the RGB bands (spectral bands) and the D band (geometric band), thereby improving the effectiveness of plane instance segmentation in a multimodal manner. Specifically, we use a dual-complexity backbone, with primarily the simpler branch learning D-band features and primarily the more complex branch learning RGB-band features. Consequently, the backbone can effectively learn D-band feature representations even when D-band training data is limited in scale, retain the powerful RGB-band feature representations of EfficientSAM, and allow the original backbone branch to be fine-tuned for the current task. To enhance the adaptability of our PlaneSAM to the RGB-D domain, we pretrain our dual-complexity backbone using the segment anything task on large-scale RGB-D data through a self-supervised pretraining strategy based on imperfect pseudo-labels. To support the segmentation of large planes, we optimize the loss function combination ratio of EfficientSAM. In addition, Faster R-CNN is used as a plane detector, and its predicted bounding boxes are fed into our dual-complexity network as prompts, thereby enabling fully automatic plane instance segmentation. Experimental results show that the proposed PlaneSAM sets a new SOTA performance on the ScanNet dataset, and outperforms previous SOTA approaches in zero-shot transfer on the 2D-3D-S, Matterport3D, and ICL-NUIM RGB-D datasets, while only incurring a 10% increase in computational overhead compared to EfficientSAM.

Fast Plane Segmentation with Line Primitives for RGB-D Sensor

Noise Model Based Line Segmentation for Plane Extraction in Sparse 3D LiDAR Data

A Fast Multiplane Segmentation Algorithm for Sparse 3-D LiDAR Point Clouds by Line Segment Grouping.

Split and Merge for Accurate Plane Segmentation in RGB-D Images

Graph-based ground segmentation of 3D LIDAR in rough area

Fast Segmentation of 3-D Point Clouds Based on Ground Plane State Tracking

Fast and Accurate Plane Segmentation of Airborne LiDAR Point Cloud Using Cross-Line Elements

An Improved RANSAC for 3D Point Cloud Plane Segmentation Based on Normal Distribution Transformation Cells

FastPlane: A Fully Convolutional Network for Real-time 3D Plane Segmentation

Noise Model-Based Line Segmentation for Plane Extraction in Sparse 3-D LiDAR Data

PlaneSegNet: Fast and Robust Plane Estimation Using a Single-stage Instance Segmentation CNN

Efficient Multi-scale Plane Extraction Based RGBD Video Segmentation

Plane Segmentation Based on the Optimal-Vector-Field in LiDAR Point Clouds

A Fast and Precise Plane Segmentation Framework for Indoor Point Clouds

A New Plane Segmentation Method of Point Cloud Based on Mean Shift and RANSAC

PlaneSAM: Multimodal Plane Instance Segmentation Using the Segment Anything Model

Fast 3D Line Segment Detection From Unorganized Point Cloud

Planar Point Cloud Segmentation Based on Edge Point Extraction and Redistribution

Line Segment Extraction for Large Scale Unorganized Point Clouds

A graph-based plane segmentation approach for noisy point clouds

3d Line Segment Detection For Unorganized Point Clouds From Multi-View Stereo