DeferredGS: Decoupled and Editable Gaussian Splatting with Deferred Shading

Tong Wu,Jia-Mu Sun,Yu-Kun Lai,Yuewen Ma,Leif Kobbelt,Lin Gao

2024-05-06

Abstract:Reconstructing and editing 3D objects and scenes both play crucial roles in computer graphics and computer vision. Neural radiance fields (NeRFs) can achieve realistic reconstruction and editing results but suffer from inefficiency in rendering. Gaussian splatting significantly accelerates rendering by rasterizing Gaussian ellipsoids. However, Gaussian splatting utilizes a single Spherical Harmonic (SH) function to model both texture and lighting, limiting independent editing capabilities of these components. Recently, attempts have been made to decouple texture and lighting with the Gaussian splatting representation but may fail to produce plausible geometry and decomposition results on reflective scenes. Additionally, the forward shading technique they employ introduces noticeable blending artifacts during relighting, as the geometry attributes of Gaussians are optimized under the original illumination and may not be suitable for novel lighting conditions. To address these issues, we introduce DeferredGS, a method for decoupling and editing the Gaussian splatting representation using deferred shading. To achieve successful decoupling, we model the illumination with a learnable environment map and define additional attributes such as texture parameters and normal direction on Gaussians, where the normal is distilled from a jointly trained signed distance function. More importantly, we apply deferred shading, resulting in more realistic relighting effects compared to previous methods. Both qualitative and quantitative experiments demonstrate the superior performance of DeferredGS in novel view synthesis and editing tasks.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper primarily aims to address the following issues:

1. **Efficient rendering and editing of 3D objects and scenes**: While Neural Radiance Fields (NeRFs) can achieve realistic reconstruction and editing results, their rendering efficiency is low. On the other hand, Gaussian splatting significantly accelerates the rendering process by rasterizing Gaussian ellipsoids but lacks the ability to independently edit textures and lighting.

2. **Independent editing of texture and lighting**: In the original Gaussian splatting representation, the appearance is approximated by a view-dependent spherical harmonic function, which mixes texture and lighting information together, limiting the ability to independently edit these two components.

3. **Geometry and decomposition results in reflective scenes**: Some recent attempts have managed to decouple texture and lighting in Gaussian splatting representations, but they may fail to produce reasonable geometry and decomposition results in reflective scenes.

4. **Aliasing artifacts during relighting**: Existing methods may exhibit significant aliasing artifacts during the relighting process because the geometric properties of Gaussian volumes are optimized under the original lighting conditions and may not be applicable under new lighting conditions.

To address the above issues, the paper proposes a method called DeferredGS, which is a decoupled Gaussian splatting representation combined with deferred shading techniques. Specifically, the main contributions of this method include:

- Defining an editable Gaussian splatting representation with decoupled geometry, texture, and lighting, where the geometric information is enhanced by extracting normals from the Signed Distance Function (SDF) to improve the accuracy of geometric reconstruction.
- Applying deferred shading techniques to Gaussian splatting representations for the first time to mitigate the aliasing artifacts present in previous methods.
- Experimental results show that DeferredGS can produce more faithful decomposition and editing results compared to previous methods.

In summary, the goal of this paper is to improve the reconstruction quality and editing capabilities of 3D scenes, especially when dealing with reflective scenes, aiming to provide an efficient solution that ensures high-quality rendering effects while supporting flexible post-editing operations.