Click-Gaussian: Interactive Segmentation to Any 3D Gaussians

Seokhun Choi,Hyeonseop Song,Jaechul Kim,Taehyeong Kim,Hoseok Do

2024-07-16

Abstract:Interactive segmentation of 3D Gaussians opens a great opportunity for real-time manipulation of 3D scenes thanks to the real-time rendering capability of 3D Gaussian Splatting. However, the current methods suffer from time-consuming post-processing to deal with noisy segmentation output. Also, they struggle to provide detailed segmentation, which is important for fine-grained manipulation of 3D scenes. In this study, we propose Click-Gaussian, which learns distinguishable feature fields of two-level granularity, facilitating segmentation without time-consuming post-processing. We delve into challenges stemming from inconsistently learned feature fields resulting from 2D segmentation obtained independently from a 3D scene. 3D segmentation accuracy deteriorates when 2D segmentation results across the views, primary cues for 3D segmentation, are in conflict. To overcome these issues, we propose Global Feature-guided Learning (GFL). GFL constructs the clusters of global feature candidates from noisy 2D segments across the views, which smooths out noises when training the features of 3D Gaussians. Our method runs in 10 ms per click, 15 to 130 times as fast as the previous methods, while also significantly improving segmentation accuracy. Our project page is available at <a class="link-external link-https" href="https://seokhunchoi.github.io/Click-Gaussian" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition,Artificial Intelligence,Graphics

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address the issue of interactive segmentation of 3D Gaussians. Specifically, the researchers propose a method called **Click-Gaussian**, which primarily aims to:

1. **Improve segmentation accuracy and efficiency**: Existing methods for 3D segmentation require time-consuming post-processing to handle noisy segmentation results and struggle to provide fine segmentation effects. Click-Gaussian achieves efficient segmentation by learning two levels of granularity features (coarse and fine) without relying on time-consuming post-processing.

2. **Resolve the consistency issue of 2D masks across different views**: Current methods suffer from decreased 3D segmentation accuracy when there are conflicts in 2D segmentation results obtained from different views. Click-Gaussian introduces a **Global Feature-guided Learning (GFL)** method, which systematically aggregates global feature candidates from the entire scene, thereby improving the consistency and reliability of feature learning.

With these improvements, Click-Gaussian can complete segmentation tasks at a speed of 10 milliseconds per click, which is 15 to 130 times faster than existing methods, while significantly enhancing segmentation accuracy.

### Summary

Click-Gaussian is an efficient and accurate interactive segmentation method for 3D Gaussians. By addressing key challenges in existing methods through two levels of granularity feature learning and global feature-guided learning, it achieves real-time segmentation and high-precision results.