SpectroMotion: Dynamic 3D Reconstruction of Specular Scenes

Cheng-De Fan,Chen-Wei Chang,Yi-Ruei Liu,Jie-Ying Lee,Jiun-Long Huang,Yu-Chee Tseng,Yu-Lun Liu

2024-10-23

Abstract:We present SpectroMotion, a novel approach that combines 3D Gaussian Splatting (3DGS) with physically-based rendering (PBR) and deformation fields to reconstruct dynamic specular scenes. Previous methods extending 3DGS to model dynamic scenes have struggled to accurately represent specular surfaces. Our method addresses this limitation by introducing a residual correction technique for accurate surface normal computation during deformation, complemented by a deformable environment map that adapts to time-varying lighting conditions. We implement a coarse-to-fine training strategy that significantly enhances both scene geometry and specular color prediction. We demonstrate that our model outperforms prior methods for view synthesis of scenes containing dynamic specular objects and that it is the only existing 3DGS method capable of synthesizing photorealistic real-world dynamic specular scenes, outperforming state-of-the-art methods in rendering complex, dynamic, and specular scenes.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the problem of reconstructing and rendering highly reflective objects in dynamic scenes. Specifically, existing 3D Gaussian Splatting (3DGS) methods face the following challenges when dealing with dynamic scenes, especially those containing specular reflective objects:

1. **Dynamic Scene Modeling**: Existing 3DGS methods struggle to accurately represent specular reflective surfaces in dynamic scenes.
2. **Time-Varying Lighting Conditions**: Existing methods cannot effectively handle lighting conditions that change over time.
3. **Geometry and Color Prediction**: Existing methods have deficiencies in predicting geometric structures and specular colors.

To overcome these challenges, the paper proposes SpectroMotion, a new method that combines 3D Gaussian Splatting, Physically-Based Rendering (PBR), and deformation fields. SpectroMotion addresses the aforementioned issues through the following innovations:

- **Residual Correction Technique**: Used to accurately compute surface normals during deformation.
- **Deformable Environment Maps**: Adapt to time-varying lighting conditions.
- **Coarse-to-Fine Training Strategy**: Significantly improves the accuracy of scene geometry and specular color predictions.

Through these innovations, SpectroMotion can more accurately reconstruct and render complex scenes containing dynamic specular reflective objects, surpassing the performance of existing methods.