MirrorGaussian: Reflecting 3D Gaussians for Reconstructing Mirror Reflections

Jiayue Liu,Xiao Tang,Freeman Cheng,Roy Yang,Zhihao Li,Jianzhuang Liu,Yi Huang,Jiaqi Lin,Shiyong Liu,Xiaofei Wu,Songcen Xu,Chun Yuan

2024-05-20

Abstract:3D Gaussian Splatting showcases notable advancements in photo-realistic and real-time novel view synthesis. However, it faces challenges in modeling mirror reflections, which exhibit substantial appearance variations from different viewpoints. To tackle this problem, we present MirrorGaussian, the first method for mirror scene reconstruction with real-time rendering based on 3D Gaussian Splatting. The key insight is grounded on the mirror symmetry between the real-world space and the virtual mirror space. We introduce an intuitive dual-rendering strategy that enables differentiable rasterization of both the real-world 3D Gaussians and the mirrored counterpart obtained by reflecting the former about the mirror plane. All 3D Gaussians are jointly optimized with the mirror plane in an end-to-end framework. MirrorGaussian achieves high-quality and real-time rendering in scenes with mirrors, empowering scene editing like adding new mirrors and objects. Comprehensive experiments on multiple datasets demonstrate that our approach significantly outperforms existing methods, achieving state-of-the-art results. Project page:

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper primarily addresses the issue of high-quality, real-time rendering in scenes containing specular reflections. Specifically, existing methods such as NeRF and 3D Gaussian Splatting have limitations when dealing with scenes that include mirrors, as these methods struggle to accurately reconstruct specular reflection effects. Specular reflections exhibit significant changes from different viewpoints, posing a challenge to reconstruction techniques based on multi-view consistency.

To solve this problem, the paper proposes MirrorGaussian, a new method based on 3D Gaussian Splatting that enables high-fidelity reconstruction and real-time rendering of scenes containing mirrors. The key idea of MirrorGaussian is to leverage the mirror symmetry between the real world and the virtual mirror space. By introducing an intuitive dual-rendering strategy, MirrorGaussian can generate realistic mirror images while maintaining efficient optimization and rendering.

Specifically, the method first obtains a rough mirror plane estimation through Structure-from-Motion (SfM) and then jointly optimizes the 3D Gaussian points and the mirror plane equation based on this estimation. To synthesize new views from arbitrary perspectives, MirrorGaussian also enhances the 3D Gaussian point cloud by adding an additional specular label to generate a specular mask.

Experimental results show that MirrorGaussian achieves significantly better performance than existing methods across multiple datasets, with improvements in quality, efficiency, and real-time rendering. Additionally, MirrorGaussian supports scene editing functions, such as adding new objects to the scene or placing new mirrors.