Shuichang Lai,Letian Huang,Jie Guo,Kai Cheng,Bowen Pan,Xiaoxiao Long,Jiangjing Lyu,Chengfei Lv,Yanwen Guo
Abstract:Reconstructing objects from posed images is a crucial and complex task in computer graphics and computer vision. While NeRF-based neural reconstruction methods have exhibited impressive reconstruction ability, they tend to be time-comsuming. Recent strategies have adopted 3D Gaussian Splatting (3D-GS) for inverse rendering, which have led to quick and effective outcomes. However, these techniques generally have difficulty in producing believable geometries and materials for glossy objects, a challenge that stems from the inherent ambiguities of inverse rendering. To address this, we introduce GlossyGS, an innovative 3D-GS-based inverse rendering framework that aims to precisely reconstruct the geometry and materials of glossy objects by integrating material priors. The key idea is the use of micro-facet geometry segmentation prior, which helps to reduce the intrinsic ambiguities and improve the decomposition of geometries and materials. Additionally, we introduce a normal map prefiltering strategy to more accurately simulate the normal distribution of reflective surfaces. These strategies are integrated into a hybrid geometry and material representation that employs both explicit and implicit methods to depict glossy objects. We demonstrate through quantitative analysis and qualitative visualization that the proposed method is effective to reconstruct high-fidelity geometries and materials of glossy objects, and performs favorably against state-of-the-arts.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: reconstructing objects with real - time materials from multi - view images, especially the challenges encountered when dealing with glossy objects in the fields of computer graphics and computer vision. Specifically, although existing methods based on Neural Radiance Field (NeRF) have demonstrated excellent reconstruction capabilities, they are usually time - consuming and have difficulty generating believable geometries and materials when dealing with glossy objects. Recently, the method of inverse rendering using 3D Gaussian Splatting (3D - GS) is fast and effective, but still has difficulties when dealing with glossy objects, mainly due to the inherent ambiguity in inverse rendering itself.
To solve these problems, the authors propose GlossyGS, an inverse - rendering framework based on 3D - GS, aiming to accurately reconstruct the geometries and materials of glossy objects by integrating material prior information. Specifically, GlossyGS introduces the following strategies:
1. **Micro - facet Geometry Segmentation Prior**:
- By using the micro - facet geometry segmentation prior, it helps to reduce the inherent ambiguity in inverse rendering, thereby improving the decomposition of geometries and materials.
- A micro - facet geometry segmentation model has been developed, which uses semantic representations derived from large - scale visual models to predict this prior.
2. **Normal Map Prefiltering Strategy**:
- The shading order has been modified. First, the normals are fused, and then shading is performed to more accurately simulate the normal distribution on the glossy surface.
- This method improves the accuracy of the normal distribution on the glossy surface.
3. **Hybrid Geometry and Material Representation**:
- Glossy objects are depicted using a combination of explicit and implicit methods. The explicit method represents the macroscopic rough geometries, while the implicit method uses feature vectors associated with these anchor points to predict multiple neural Gaussian volumes.
- Neural Gaussian volumes are crucial for accurately simulating the normal distribution and material properties of micro - facets, providing a detailed and realistic representation of the glossy object surface.
Through these improvements, GlossyGS can demonstrate its effectiveness in both quantitative analysis and qualitative visualization, successfully reconstructing high - fidelity geometries and materials, especially for glossy objects, and outperforming existing methods in performance.
### Formula Summary
- **α - blending Weights**:
\[
w_k=\alpha_k\prod_{j = 1}^{k - 1}(1-\alpha_j),\quad\text{where}\quad\alpha = o\cdot g
\]
- **Projected Gaussian Function**:
\[
g(u, v)=e^{-\frac{1}{2}\left([u, v]^\top-\mu'\right)^\top\Sigma'^{-1}\left([u, v]^\top-\mu'\right)}
\]
- **Pixel Color Calculation**:
\[
C(u, v)=\sum_{k = 1}^N SH_k(u, v)w_k
\]
- **BRDF Model**:
\[
f(\omega_i,\omega_o)=\frac{(1 - m)a}{\pi}+\frac{DFG}{4(\omega_i\cdot n)(\omega_o\cdot n)}
\]
- **Smooth Regularization Loss**:
\[
L_s=\|F_m - d(f_{\text{micro}})-F_m - d(f_{\text{micro}}+\xi)\|_1
\]
These formulas ensure the efficiency and accuracy of GlossyGS when dealing with the inverse rendering of glossy objects.