TexGaussian: Generating High-quality PBR Material via Octree-based 3D Gaussian Splatting

Bojun Xiong,Jialun Liu,Jiakui Hu,Chenming Wu,Jinbo Wu,Xing Liu,Chen Zhao,Errui Ding,Zhouhui Lian

2024-11-29

Abstract:Physically Based Rendering (PBR) materials play a crucial role in modern graphics, enabling photorealistic rendering across diverse environment maps. Developing an effective and efficient algorithm that is capable of automatically generating high-quality PBR materials rather than RGB texture for 3D meshes can significantly streamline the 3D content creation. Most existing methods leverage pre-trained 2D diffusion models for multi-view image synthesis, which often leads to severe inconsistency between the generated textures and input 3D meshes. This paper presents TexGaussian, a novel method that uses octant-aligned 3D Gaussian Splatting for rapid PBR material generation. Specifically, we place each 3D Gaussian on the finest leaf node of the octree built from the input 3D mesh to render the multiview images not only for the albedo map but also for roughness and metallic. Moreover, our model is trained in a regression manner instead of diffusion denoising, capable of generating the PBR material for a 3D mesh in a single feed-forward process. Extensive experiments on publicly available benchmarks demonstrate that our method synthesizes more visually pleasing PBR materials and runs faster than previous methods in both unconditional and text-conditional scenarios, which exhibit better consistency with the given geometry. Our code and trained models are available at <a class="link-external link-https" href="https://3d-aigc.github.io/TexGaussian" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **How to automatically generate high - quality physically - based rendering (PBR) materials to achieve realistic rendering of 3D models in different lighting environments?**

Specifically, existing methods mainly rely on pre - trained 2D diffusion models for multi - view image synthesis, which often leads to serious inconsistencies between the generated textures and the input 3D models. In addition, the assets generated by these methods often contain baked - in lighting effects, which will significantly reduce the quality of the final rendering when placed in new lighting conditions. Meanwhile, although the method of training neural networks directly on 3D data can maintain global consistency, due to the discreteness and sparsity of point cloud representations, the generated textures are blurry and not fine - grained enough.

To solve these problems, this paper proposes **TexGaussian**, a method based on octree - aligned 3D Gaussian Splatting, which can directly generate high - quality PBR materials in 3D space and maintain 3D global consistency. This method predicts PBR material parameters by regression, avoiding multi - view sampling and score distillation sampling in the diffusion denoising process, thus achieving a faster generation speed.

### Main contributions:
1. **Proposed an octree - aligned 3D Gaussian Splatting method** for generating high - quality PBR materials on untextured 3D meshes, making full use of the supervision information of 2D images and avoiding blurry results caused by the discreteness of 3D point clouds.
2. **Adopted a regression - based method to train the 3D U - Net model** instead of diffusion denoising, achieving a faster generation speed.
3. **Proposed TexGaussian**, a new PBR material generation method based on the above two techniques, which directly generates PBR materials in 3D space for the first time and verifies its superiority in quality and efficiency through qualitative and quantitative experiments.

Through these innovations, TexGaussian can generate PBR materials that are highly consistent with the input geometry, of higher quality, and perform well in different lighting environments.