Neural Surface Priors for Editable Gaussian Splatting

Jakub Szymkowiak,Weronika Jakubowska,Dawid Malarz,Weronika Smolak-Dyżewska,Maciej Zięba,Przemysław Musialski,Wojtek Pałubicki,Przemysław Spurek

2024-11-27

Abstract:In computer graphics, there is a need to recover easily modifiable representations of 3D geometry and appearance from image data. We introduce a novel method for this task using 3D Gaussian Splatting, which enables intuitive scene editing through mesh adjustments. Starting with input images and camera poses, we reconstruct the underlying geometry using a neural Signed Distance Field and extract a high-quality mesh. Our model then estimates a set of Gaussians, where each component is flat, and the opacity is conditioned on the recovered neural surface. To facilitate editing, we produce a proxy representation that encodes information about the Gaussians' shape and position. Unlike other methods, our pipeline allows modifications applied to the extracted mesh to be propagated to the proxy representation, from which we recover the updated parameters of the Gaussians. This effectively transfers the mesh edits back to the recovered appearance representation. By leveraging mesh-guided transformations, our approach simplifies 3D scene editing and offers improvements over existing methods in terms of usability and visual fidelity of edits. The complete source code for this project can be accessed at \url{<a class="link-external link-https" href="https://github.com/WJakubowska/NeuralSurfacePriors" rel="external noopener nofollow">this https URL</a>}

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to solve the problem of recovering 3D geometric structures and appearance representations from image data, especially in order to enable convenient modification of the reconstructed scene. Specifically, the paper proposes a new method using 3D Gaussian Splatting technology, allowing users to intuitively modify the appearance of the scene by editing the extracted mesh model.

#### Main problems:
1. **Insufficiency of existing 3D scene reconstruction methods**: Current methods such as NeRF and 3D Gaussian Splatting, although performing well in visual fidelity, cannot provide a connected surface mesh, which limits the convenient modification of the reconstructed scene.
2. **Lack of effective editing tools**: Traditional 3D scene reconstruction methods do not support direct propagation to the appearance representation through mesh editing, resulting in the edited mesh being inconsistent with the original appearance.
3. **Improving editing flexibility and visual fidelity**: A method is needed that allows users to easily modify 3D scenes while maintaining high visual quality.

#### Solutions:
- **Introducing Neural Surface Priors**: Utilize pre - trained neural networks to generate high - quality mesh models and combine these meshes with Gaussian Splatting to achieve efficient appearance editing.
- **Jointly optimizing the mesh and Gaussian Splatting**: By editing the mesh, these modifications can be propagated to the Gaussian Splatting, thus updating the appearance representation of the scene.
- **Using Proxy Triangle Soup**: Through this intermediate representation form, the changes in mesh editing can be passed to the Gaussian Splatting, ensuring that the edited appearance is consistent with the mesh.

#### Method overview:
1. **Obtaining Neural Surface Priors**: Use PermutoSDF to generate a neural SDF model, guide the transparency of the Gaussian kernel and extract the mesh for subsequent editing.
2. **Recovering appearance**: Train a 3D Gaussian Splatting - based model, condition the transparency of the Gaussian kernel according to the distance of the neural surface, and ensure that each Gaussian kernel is tiled on the surface.
3. **Mesh - guided appearance modification**: Encode the position and shape of the Gaussian kernel into a triangle - soup proxy structure. After manually adjusting the mesh, propagate these modifications to the proxy structure and finally update the position and orientation of the Gaussian kernel.

Through this method, the paper achieves an easily editable 3D scene representation while maintaining high visual fidelity.