Abstract:We introduce FabricDiffusion, a method for transferring fabric textures from a single clothing image to 3D garments of arbitrary shapes. Existing approaches typically synthesize textures on the garment surface through 2D-to-3D texture mapping or depth-aware inpainting via generative models. Unfortunately, these methods often struggle to capture and preserve texture details, particularly due to challenging occlusions, distortions, or poses in the input image. Inspired by the observation that in the fashion industry, most garments are constructed by stitching sewing patterns with flat, repeatable textures, we cast the task of clothing texture transfer as extracting distortion-free, tileable texture materials that are subsequently mapped onto the UV space of the garment. Building upon this insight, we train a denoising diffusion model with a large-scale synthetic dataset to rectify distortions in the input texture image. This process yields a flat texture map that enables a tight coupling with existing Physically-Based Rendering (PBR) material generation pipelines, allowing for realistic relighting of the garment under various lighting conditions. We show that FabricDiffusion can transfer various features from a single clothing image including texture patterns, material properties, and detailed prints and logos. Extensive experiments demonstrate that our model significantly outperforms state-to-the-art methods on both synthetic data and real-world, in-the-wild clothing images while generalizing to unseen textures and garment shapes.

What problem does this paper attempt to address?

### Problems the paper attempts to solve The paper "FabricDiffusion: High - Fidelity Texture Transfer for 3D Garments Generation from In - The - Wild Clothing Images" attempts to solve the problem of extracting high - quality textures from 2D clothing images in the real world and transferring them to 3D clothing models. Specifically, the paper aims to: 1. **High - fidelity texture transfer**: Extract high - quality texture maps and printed patterns from a single 2D clothing image and transfer them without distortion to 3D clothing models of arbitrary shapes. 2. **Handle complex textures and materials**: Be able to handle different types of textures, patterns and materials, and generate texture maps that include not only diffuse albedo, but also roughness, normal and metallic. 3. **Photorealistic rendering**: The generated 3D clothing can be accurately relit and rendered under various lighting conditions, thus achieving photorealistic virtual fitting and e - commerce applications. ### Solutions To achieve the above goals, the paper proposes the **FabricDiffusion** method, and its main contributions include: 1. **Texture extraction and generation**: - **Texture normalization**: Train a denoising diffusion model to correct geometric distortions in the input texture image and generate flat and seamless texture maps. - **Multi - view consistency**: Use a synthetic dataset to train the model to ensure that the generated textures are consistent from different perspectives. 2. **Synthetic dataset**: - **Large - scale synthetic data**: Construct a large - scale synthetic dataset containing more than 100,000 textile color images, 3,800 PBR material texture maps, 7,000 printed patterns (such as logos), and 22 original 3D clothing models. - **Photorealistic rendering**: Use PBR technology to render these textures and printed patterns under different lighting and environmental conditions to simulate real - world scenes. 3. **PBR material generation**: - **Compatible with existing pipelines**: The generated normalized textures can be seamlessly integrated into existing physically - based rendering (PBR) material estimation pipelines, such as MatFusion. - **High - quality material maps**: By fine - tuning the pre - trained MatFusion model, generate high - quality PBR material maps for photorealistic 3D clothing rendering. 4. **Experimental verification**: - **Synthetic data and real - world data**: Conduct extensive experiments on synthetic data and real - world images to verify the performance of the model. - **Zero - shot generalization**: Although trained entirely on synthetic data, the model can achieve zero - shot generalization on real - world images and handle complex textures and printed patterns. ### Summary FabricDiffusion proposes an innovative method to extract high - quality textures and printed patterns from a single 2D clothing image and transfer them without distortion to 3D clothing models through a denoising diffusion model and a large - scale synthetic dataset. This method performs well in handling complex textures and materials and can generate high - quality PBR material maps, which are suitable for application scenarios such as virtual fitting, e - commerce and games.

FabricDiffusion: High-Fidelity Texture Transfer for 3D Garments Generation from In-The-Wild Clothing Images

Image Reference-guided Fashion Design with Structure-aware Transfer by Diffusion Models.

DiffCloth: Diffusion Based Garment Synthesis and Manipulation via Structural Cross-modal Semantic Alignment

Cloth2Tex: A Customized Cloth Texture Generation Pipeline for 3D Virtual Try-On

Toward Accurate and Realistic Garment Texture Transfer with Attention to Details

IGR: Improving Diffusion Model for Garment Restoration from Person Image

Texture-Preserving Diffusion Models for High-Fidelity Virtual Try-On

Modeling and Realization of Image-Based Garment Texture Transfer

FashionR2R: Texture-preserving Rendered-to-Real Image Translation with Diffusion Models

DiffFashion: Reference-based Fashion Design with Structure-aware Transfer by Diffusion Models

WarpDiffusion: Efficient Diffusion Model for High-Fidelity Virtual Try-on

Spatial-Aware Texture Transformer for High-Fidelity Garment Transfer.

Garment3DGen: 3D Garment Stylization and Texture Generation

RAGDiffusion: Faithful Cloth Generation via External Knowledge Assimilation

Improving Virtual Try-On with Garment-focused Diffusion Models

GarmentDreamer: 3DGS Guided Garment Synthesis with Diverse Geometry and Texture Details

Enhancing consistency in virtual try-on: A novel diffusion-based approach

Shape-preserving Swapping Autoencoder for Garment Texture Transfer

Taming the Power of Diffusion Models for High-Quality Virtual Try-On with Appearance Flow

AnyDressing: Customizable Multi-Garment Virtual Dressing via Latent Diffusion Models

Woven Fabric Capture with a Reflection-Transmission Photo Pair