Segment Any 3D Gaussians

Jiazhong Cen,Jiemin Fang,Chen Yang,Lingxi Xie,Xiaopeng Zhang,Wei Shen,Qi Tian

2024-05-27

Abstract:This paper presents SAGA (Segment Any 3D GAussians), a highly efficient 3D promptable segmentation method based on 3D Gaussian Splatting (3D-GS). Given 2D visual prompts as input, SAGA can segment the corresponding 3D target represented by 3D Gaussians within 4 ms. This is achieved by attaching an scale-gated affinity feature to each 3D Gaussian to endow it a new property towards multi-granularity segmentation. Specifically, a scale-aware contrastive training strategy is proposed for the scale-gated affinity feature learning. It 1) distills the segmentation capability of the Segment Anything Model (SAM) from 2D masks into the affinity features and 2) employs a soft scale gate mechanism to deal with multi-granularity ambiguity in 3D segmentation through adjusting the magnitude of each feature channel according to a specified 3D physical scale. Evaluations demonstrate that SAGA achieves real-time multi-granularity segmentation with quality comparable to state-of-the-art methods. As one of the first methods addressing promptable segmentation in 3D-GS, the simplicity and effectiveness of SAGA pave the way for future advancements in this field. Our code will be released.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the problem of promptable segmentation in 3D Gaussian Splatting (3D-GS). Specifically, the research team proposes a method called SAGA (Segment Any 3D GAussians), which can efficiently perform multi-granularity 3D object segmentation. SAGA attaches scale-gated affinity features to each 3D Gaussian point, enabling these Gaussian points to have segmentation capabilities. Additionally, to handle multi-granularity ambiguity, SAGA employs a soft scale-gating mechanism to adjust the amplitude of different feature channels, thereby addressing the issue of the same Gaussian point potentially belonging to different parts or objects at different physical scales.

Experimental results show that SAGA can complete real-time multi-granularity segmentation within a few milliseconds, and the segmentation quality is comparable to existing state-of-the-art methods. Furthermore, since SAGA seamlessly integrates segmentation capabilities into 3D-GS, it is highly efficient and can segment each 3D object within 4 milliseconds when given 2D visual prompts. The research team believes that the simplicity and effectiveness of SAGA pave the way for future promptable segmentation in 3D-GS.