Gradient-Weighted Feature Back-Projection: A Fast Alternative to Feature Distillation in 3D Gaussian Splatting

Joji Joseph,Bharadwaj Amrutur,Shalabh Bhatnagar

2024-11-19

Abstract:We introduce a training-free method for feature field rendering in Gaussian splatting. Our approach back-projects 2D features into pre-trained 3D Gaussians, using a weighted sum based on each Gaussian's influence in the final rendering. While most training-based feature field rendering methods excel at 2D segmentation but perform poorly at 3D segmentation without post-processing, our method achieves high-quality results in both 2D and 3D segmentation. Experimental results demonstrate that our approach is fast, scalable, and offers performance comparable to training-based methods.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in 3D Gaussian Splatting, although the existing training - based methods perform well in 2D segmentation tasks, they perform poorly in 3D segmentation tasks and are computationally expensive. Specifically:

1. **3D Segmentation Accuracy Problem**: Traditional training - based feature field rendering methods are ineffective in handling 3D segmentation because these methods usually require a large number of post - processing steps to correct the segmentation results.

2. **Computational Efficiency Problem**: Directly training the Gaussian Splatting model to render feature maps is very time - consuming, especially when the feature space dimension is high. This limits the feasibility of these methods in practical applications.

To solve these problems, the author proposes a training - free method - Gradient - Weighted Feature Back - Projection. This method realizes efficient and high - quality 2D and 3D segmentation by back - projecting 2D features into the pre - trained 3D Gaussian distribution and using the influence of each Gaussian distribution on the final rendering for weighted summation.

### Specific Problem Description

- **Inaccurate 3D Segmentation**: Since the rendered features are composed of the weighted sum of multiple Gaussian distributions, the features of a single Gaussian distribution cannot be directly mapped to the finally rendered feature map, resulting in lower 3D segmentation accuracy.

- **High Computational Complexity**: Training the Gaussian Splatting model to directly generate feature maps requires a large amount of computational resources and time, especially when dealing with high - dimensional feature spaces.

### Proposed Solutions

- **Gradient - Weighted Feature Back - Projection**: By using the gradient back - propagation during inference, calculate the contribution of each Gaussian distribution to the finally rendered features, thereby achieving efficient feature aggregation and back - projection.

- **Fast and Scalable**: This method does not require an additional training process and can significantly improve computational efficiency while maintaining high - quality segmentation results.

Through this method, the author not only improves the accuracy of 3D segmentation but also significantly reduces the computational time and resource consumption, making this method more feasible and practical in practical applications.