Gaussian Splatting in Mirrors: Reflection-Aware Rendering via Virtual Camera Optimization

Zihan Wang,Shuzhe Wang,Matias Turkulainen,Junyuan Fang,Juho Kannala

2024-10-02

Abstract:Recent advancements in 3D Gaussian Splatting (3D-GS) have revolutionized novel view synthesis, facilitating real-time, high-quality image rendering. However, in scenarios involving reflective surfaces, particularly mirrors, 3D-GS often misinterprets reflections as virtual spaces, resulting in blurred and inconsistent multi-view rendering within mirrors. Our paper presents a novel method aimed at obtaining high-quality multi-view consistent reflection rendering by modelling reflections as physically-based virtual cameras. We estimate mirror planes with depth and normal estimates from 3D-GS and define virtual cameras that are placed symmetrically about the mirror plane. These virtual cameras are then used to explain mirror reflections in the scene. To address imperfections in mirror plane estimates, we propose a straightforward yet effective virtual camera optimization method to enhance reflection quality. We collect a new mirror dataset including three real-world scenarios for more diverse evaluation. Experimental validation on both Mirror-Nerf and our real-world dataset demonstrate the efficacy of our approach. We achieve comparable or superior results while significantly reducing training time compared to previous state-of-the-art.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to solve the problems encountered in new view synthesis by the 3D Gaussian Splatting (3D - GS) method in scenes containing reflective surfaces such as mirrors. Specifically:

1. **Problem Description**:
- When dealing with scenes with reflective surfaces (such as mirrors), the existing 3D - GS method is prone to misidentifying reflections as virtual spaces, resulting in blurry and inconsistent multi - view rendering in the mirror.
- Such misjudgment makes the rendering of the mirror and its edges blurry and chaotic, affecting the overall quality of new view synthesis.

2. **Solution**:
- The paper proposes a new method to achieve high - quality multi - view consistent reflection rendering by modeling reflections as physics - based virtual cameras.
- Specific steps include:
- **Mirror Plane Estimation**: Use the depth and normal estimates generated by 3D - GS to predict the mirror plane equation.
- **Virtual Camera Definition**: Symmetrically place virtual cameras according to the predicted mirror plane equation.
- **Virtual Camera Optimization**: To deal with the imperfections in mirror plane estimation, a simple virtual camera pose optimization method is proposed to improve the reflection quality.
- **Dataset Collection**: A new dataset containing three real - world scenes is collected, covering mirrors of different scales and shapes, to provide a broader evaluation environment.

3. **Contributions**:
- A rendering method based on 3D - GS is proposed, which can handle scenes containing mirror reflections and maintain real - time rendering capabilities.
- A real - world dataset containing mirrors of various scales and sizes is collected to evaluate the effectiveness of the method.
- Experimental results show that this method matches or exceeds the existing state - of - the - art methods on multiple metrics while retaining the efficient rendering quality of 3D - GS.

4. **Formula Presentation**:
- **Pixel Color Calculation**:
\[
\hat{C}=\sum_{i \in N} c_i \alpha_i T_i, \quad \text{where} \quad T_i=\prod_{j = 1}^{i - 1}(1-\alpha_j)
\]
- **Gaussian Mixture Term**:
\[
\alpha_i=O_i \cdot \exp\left(\frac{1}{2}(p-\hat{\mu}_i)\Sigma_i^{-1}(p - \hat{\mu}_i)\right)
\]
- **Depth Estimation**:
\[
\hat{D}=\sum_{i \in N} d_i \alpha_i \prod_{j = 1}^{i - 1}(1-\alpha_j)
\]

Through the above methods, the paper successfully solves the challenges encountered by 3D - GS when dealing with reflective surfaces, improving the quality and efficiency of new view synthesis.