Rui Zhang,Tianyue Luo,Weidong Yang,Ben Fei,Jingyi Xu,Qingyuan Zhou,Keyi Liu,Ying He
Abstract:3D Gaussian Splatting (3D-GS) has made a notable advancement in the field of neural rendering, 3D scene reconstruction, and novel view synthesis. Nevertheless, 3D-GS encounters the main challenge when it comes to accurately representing physical reflections, especially in the case of total reflection and semi-reflection that are commonly found in real-world scenes. This limitation causes reflections to be mistakenly treated as independent elements with physical presence, leading to imprecise reconstructions. Herein, to tackle this challenge, we propose RefGaussian to disentangle reflections from 3D-GS for realistically modeling reflections. Specifically, we propose to split a scene into transmitted and reflected components and represent these components using two Spherical Harmonics (SH). Given that this decomposition is not fully determined, we employ local regularization techniques to ensure local smoothness for both the transmitted and reflected components, thereby achieving more plausible decomposition outcomes than 3D-GS. Experimental results demonstrate that our approach achieves superior novel view synthesis and accurate depth estimation outcomes. Furthermore, it enables the utilization of scene editing applications, ensuring both high-quality results and physical coherence.
What problem does this paper attempt to address?
### Problems Addressed by the Paper
The paper aims to address the main challenges encountered in 3D Gaussian Splatting (3D-GS) in neural rendering, 3D scene reconstruction, and novel view synthesis—specifically, the inability to accurately represent physical reflections, especially in cases of full and partial reflections commonly found in real-world scenarios. Existing methods such as NeRF and its variants (including 3D-GS) have shortcomings when dealing with reflective surfaces (e.g., mirrors, glass, screens). These models often misinterpret reflective content as independent entities, leading to a decrease in the fidelity and coherence of the rendered scenes.
To solve this problem, the paper proposes RefGaussian, a new method for separating reflections from 3D-GS. Specifically, RefGaussian decomposes the scene into transmission and reflection parts and uses two sets of Spherical Harmonics (SH) to represent these parts separately. Additionally, to ensure the rationality of the decomposition process, the paper introduces local regularization techniques to ensure the smoothness of both the transmission and reflection parts. Experimental results show that this method not only performs well in novel view synthesis and depth estimation but also supports high-quality and physically coherent scene editing applications.
### Main Contributions
1. **Decomposition Method for Reflection Modeling**: A new joint training and synchronous rendering scheme is proposed without the need for additional Gaussian spheres. By adding three reflection-related parameters (reflection spherical harmonics, reflection opacity, reflection confidence), efficient and realistic reflection modeling is achieved.
2. **Bilateral Smoothness Constraint**: Bilateral smoothness constraints and reflection map smoothness constraints are introduced, considering depth changes and color differences, to collaboratively optimize and promote scene decomposition.
3. **Performance Validation**: RefGaussian demonstrates superior performance in view synthesis on real datasets containing reflections and is competitive in a broader range of scenarios. Additionally, it shows potential for reflection operations in scene editing tasks, such as highlight and shadow adjustments.