Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields

Shijie Zhou,Haoran Chang,Sicheng Jiang,Zhiwen Fan,Zehao Zhu,Dejia Xu,Pradyumna Chari,Suya You,Zhangyang Wang,Achuta Kadambi

2024-04-08

Abstract:3D scene representations have gained immense popularity in recent years. Methods that use Neural Radiance fields are versatile for traditional tasks such as novel view synthesis. In recent times, some work has emerged that aims to extend the functionality of NeRF beyond view synthesis, for semantically aware tasks such as editing and segmentation using 3D feature field distillation from 2D foundation models. However, these methods have two major limitations: (a) they are limited by the rendering speed of NeRF pipelines, and (b) implicitly represented feature fields suffer from continuity artifacts reducing feature quality. Recently, 3D Gaussian Splatting has shown state-of-the-art performance on real-time radiance field rendering. In this work, we go one step further: in addition to radiance field rendering, we enable 3D Gaussian splatting on arbitrary-dimension semantic features via 2D foundation model distillation. This translation is not straightforward: naively incorporating feature fields in the 3DGS framework encounters significant challenges, notably the disparities in spatial resolution and channel consistency between RGB images and feature maps. We propose architectural and training changes to efficiently avert this problem. Our proposed method is general, and our experiments showcase novel view semantic segmentation, language-guided editing and segment anything through learning feature fields from state-of-the-art 2D foundation models such as SAM and CLIP-LSeg. Across experiments, our distillation method is able to provide comparable or better results, while being significantly faster to both train and render. Additionally, to the best of our knowledge, we are the first method to enable point and bounding-box prompting for radiance field manipulation, by leveraging the SAM model. Project website at: <a class="link-external link-https" href="https://feature-3dgs.github.io/" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The main goal of this paper is to address the limitations of existing 3D scene representation methods in terms of functionality and efficiency. Specifically, the paper attempts to solve the following key issues:

1. **Enhancing the functionality of 3D scene representation**: Traditional 3D scene representation methods like NeRF mainly focus on novel view synthesis, but their functionality is relatively limited. The paper enhances 3D Gaussian Splatting by integrating large 2D foundational models, enabling it to perform not only novel view synthesis but also tasks such as semantic segmentation and language-guided editing.

2. **Addressing slow rendering speed**: Methods based on NeRF are slow during training and inference. By adopting 3D Gaussian Splatting technology, the proposed method in the paper significantly improves speed while maintaining high-quality rendering.

3. **Improving feature quality**: Implicit representation feature fields have continuity artifacts that reduce feature quality. The paper addresses this issue by learning a structured low-dimensional feature field and then upsampling it using a lightweight convolutional decoder.

4. **Achieving efficient feature field distillation**: The paper proposes a new framework for efficiently distilling feature fields from 2D foundational models and applying them to the 3D Gaussian Splatting framework. This method not only speeds up feature field distillation but also improves the quality of feature rendering.

In summary, this research aims to overcome the limitations of existing 3D scene representation methods by combining 3D Gaussian Splatting and feature field distillation techniques in a novel way, resulting in faster, more flexible, and more powerful 3D scene representations.