StableMaterials: Enhancing Diversity in Material Generation via Semi-Supervised Learning

Giuseppe Vecchio

2024-07-28

Abstract:We introduce StableMaterials, a novel approach for generating photorealistic physical-based rendering (PBR) materials that integrate semi-supervised learning with Latent Diffusion Models (LDMs). Our method employs adversarial training to distill knowledge from existing large-scale image generation models, minimizing the reliance on annotated data and enhancing the diversity in generation. This distillation approach aligns the distribution of the generated materials with that of image textures from an SDXL model, enabling the generation of novel materials that are not present in the initial training dataset. Furthermore, we employ a diffusion-based refiner model to improve the visual quality of the samples and achieve high-resolution generation. Finally, we distill a latent consistency model for fast generation in just four steps and propose a new tileability technique that removes visual artifacts typically associated with fewer diffusion steps. We detail the architecture and training process of StableMaterials, the integration of semi-supervised training within existing LDM frameworks and show the advantages of our approach. Comparative evaluations with state-of-the-art methods show the effectiveness of StableMaterials, highlighting its potential applications in computer graphics and beyond. StableMaterials is publicly available at <a class="link-external link-https" href="https://gvecchio.com/stablematerials" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The paper attempts to address the challenges faced in generating realistic Physically Based Rendering (PBR) materials in computer graphics, particularly in improving the diversity and quality of generation in the context of scarce annotated data. Specifically, the paper introduces a new method called StableMaterials, which leverages unannotated data through semi-supervised learning combined with adversarial training to enhance the diversity and realism of generated materials. Additionally, the method proposes a novel "feature rolling" technique to achieve seamless tiling and accelerates the generation process by distilling a latent consistency model. These improvements enable StableMaterials to generate high-resolution and high-quality PBR materials with fewer diffusion steps, while overcoming the limitations of existing methods in representing complex materials and the shortcomings of relying on image generation models. Overall, StableMaterials aims to enhance the quality and diversity of material generation by combining supervised and unsupervised learning strategies, especially in scenarios with limited annotated data.