Feature Splatting for Better Novel View Synthesis with Low Overlap

T. Berriel Martins,Javier Civera

2024-09-30

Abstract:3D Gaussian Splatting has emerged as a very promising scene representation, achieving state-of-the-art quality in novel view synthesis significantly faster than competing alternatives. However, its use of spherical harmonics to represent scene colors limits the expressivity of 3D Gaussians and, as a consequence, the capability of the representation to generalize as we move away from the training views. In this paper, we propose to encode the color information of 3D Gaussians into per-Gaussian feature vectors, which we denote as Feature Splatting (FeatSplat). To synthesize a novel view, Gaussians are first "splatted" into the image plane, then the corresponding feature vectors are alpha-blended, and finally the blended vector is decoded by a small MLP to render the RGB pixel values. To further inform the model, we concatenate a camera embedding to the blended feature vector, to condition the decoding also on the viewpoint information. Our experiments show that these novel model for encoding the radiance considerably improves novel view synthesis for low overlap views that are distant from the training views. Finally, we also show the capacity and convenience of our feature vector representation, demonstrating its capability not only to generate RGB values for novel views, but also their per-pixel semantic labels. Code available at <a class="link-external link-https" href="https://github.com/tberriel/FeatSplat" rel="external noopener nofollow">this https URL</a> . Keywords: Gaussian Splatting, Novel View Synthesis, Feature Splatting

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the problem of novel view synthesis in 3D scene representation, particularly for low-overlap view synthesis outside of training views. Specifically, existing 3D Gaussian Splatting methods use spherical harmonics to represent scene colors, which limits the expressive power of 3D Gaussian point clouds and results in poor performance on novel views far from the training perspectives. The paper proposes a new method—Feature Splatting, which replaces spherical harmonics with feature vectors to encode the color information of each 3D Gaussian point. This approach improves the quality of novel view synthesis, especially in cases with low overlap with training views. Additionally, the method can generate semantic labels for new views, demonstrating its flexibility in encoding different types of information. Experimental results show that Feature Splatting outperforms existing methods on both standard datasets and the challenging ScanNet++ dataset.