Jan-Niklas Dihlmann,Arjun Majumdar,Andreas Engelhardt,Raphael Braun,Hendrik P.A. Lensch
Abstract:3D reconstruction and relighting of objects made from scattering materials present a significant challenge due to the complex light transport beneath the surface. 3D Gaussian Splatting introduced high-quality novel view synthesis at real-time speeds. While 3D Gaussians efficiently approximate an object's surface, they fail to capture the volumetric properties of subsurface scattering. We propose a framework for optimizing an object's shape together with the radiance transfer field given multi-view OLAT (one light at a time) data. Our method decomposes the scene into an explicit surface represented as 3D Gaussians, with a spatially varying BRDF, and an implicit volumetric representation of the scattering component. A learned incident light field accounts for shadowing. We optimize all parameters jointly via ray-traced differentiable rendering. Our approach enables material editing, relighting and novel view synthesis at interactive rates. We show successful application on synthetic data and introduce a newly acquired multi-view multi-light dataset of objects in a light-stage setup. Compared to previous work we achieve comparable or better results at a fraction of optimization and rendering time while enabling detailed control over material attributes. Project page <a class="link-external link-https" href="https://sss.jdihlmann.com/" rel="external noopener nofollow">this https URL</a>
What problem does this paper attempt to address?
The problem that this paper attempts to solve is how to efficiently and high - quality simulate the subsurface scattering (SSS) effect in 3D reconstruction and relighting. Specifically, although the traditional 3D Gaussian Splatting method can generate high - quality novel - view - synthesis images at real - time speed, it mainly focuses on the approximation of the object surface and cannot capture the volume properties of subsurface scattering, which leads to unnatural representations of materials such as wax, marble, and skin. Therefore, this paper proposes a new framework, aiming to optimize the radiation transfer field while optimizing the object shape, in order to achieve a more realistic subsurface scattering effect. This framework can decompose the scene into an explicit surface representation (using 3D Gaussian representation and having spatially - varying BRDF) and an implicit volume - scattering - component representation. By jointly optimizing all parameters, this method can achieve functions such as material editing, relighting, and novel - view - synthesis while maintaining real - time performance.
### Main contributions:
1. **Hybrid representation**: Extend the 3D Gaussian Splatting method, combine the explicit surface - appearance model and the implicit subsurface - scattering model to more accurately simulate the subsurface scattering effect.
2. **Lightweight residual - prediction network**: Introduce a lightweight MLP to learn the shadow component of subsurface scattering and predict the incident light at the same time, in order to enhance the neural representation of local and global light - transfer.
3. **Deferred shading**: Perform shading in the image space to improve the representation of high - frequency details (such as specular reflection).
4. **Multi - view multi - light - source dataset**: Create a new multi - view multi - light - source dataset, including synthetic and real - world objects, for training and validating the model.
### Method overview:
- **3D Gaussian Splatting**: Use 3D Gaussian to represent the geometry and appearance of the scene, and achieve fast 3D reconstruction and novel - view - synthesis through efficient splatting techniques.
- **Subsurface - scattering modeling**: Introduce an implicit neural network to estimate the outgoing radiation of each 3D Gaussian under the illumination of a single light source, thereby approximating the internal light - transfer.
- **Joint optimization**: Through differentiable - rendering techniques, jointly optimize the geometry, direct reflection, and global subsurface - scattering effect.
- **Deferred shading**: Calculate the BRDF model in the image space to capture high - frequency details, such as specular reflection.
### Experimental results:
- **Qualitative results**: Show the novel - view - synthesis and relighting effects on synthetic and real - world datasets, and can clearly capture subsurface scattering and specular reflection.
- **Quantitative results**: Evaluate the novel - view - synthesis task on synthetic and real - world datasets, using metrics such as PSNR, SSIM, and LPIPS. The results show that this method achieves high - quality results while maintaining real - time performance.
In general, by extending the 3D Gaussian Splatting method, this paper successfully solves the problem of efficient simulation of subsurface scattering, providing a new solution for real - time 3D reconstruction and relighting.