FashionR2R: Texture-preserving Rendered-to-Real Image Translation with Diffusion Models

Rui Hu,Qian He,Gaofeng He,Jiedong Zhuang,Huang Chen,Huafeng Liu,Huamin Wang

2024-10-18

Abstract:Modeling and producing lifelike clothed human images has attracted researchers' attention from different areas for decades, with the complexity from highly articulated and structured content. Rendering algorithms decompose and simulate the imaging process of a camera, while are limited by the accuracy of modeled variables and the efficiency of computation. Generative models can produce impressively vivid human images, however still lacking in controllability and editability. This paper studies photorealism enhancement of rendered images, leveraging generative power from diffusion models on the controlled basis of rendering. We introduce a novel framework to translate rendered images into their realistic counterparts, which consists of two stages: Domain Knowledge Injection (DKI) and Realistic Image Generation (RIG). In DKI, we adopt positive (real) domain finetuning and negative (rendered) domain embedding to inject knowledge into a pretrained Text-to-image (T2I) diffusion model. In RIG, we generate the realistic image corresponding to the input rendered image, with a Texture-preserving Attention Control (TAC) to preserve fine-grained clothing textures, exploiting the decoupled features encoded in the UNet structure. Additionally, we introduce SynFashion dataset, featuring high-quality digital clothing images with diverse textures. Extensive experimental results demonstrate the superiority and effectiveness of our method in rendered-to-real image translation.

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to convert computer - rendered fashion images into real fashion images while preserving the detailed textures of clothing. Specifically, existing methods have limitations in enhancing the realism of rendered images. For example, they may not be able to use real - image resources for highly detailed enhancement, or they may be unstable and suffer from mode collapse during adversarial training. In addition, these methods also have difficulties in maintaining the fine - grained textures in clothing images. Therefore, this paper proposes a new framework based on the diffusion model to solve these problems. It aims to achieve the conversion from rendered images to real images by leveraging the generative ability of pre - trained text - to - image diffusion models and combining the guidance of rendering prior knowledge. This framework consists of two stages: Domain Knowledge Injection (DKI) and Realistic Image Generation (RIG). In the DKI stage, knowledge is injected into the model through forward domain fine - tuning and negative domain embedding of the pre - trained text - to - image diffusion model. In the RIG stage, Texture - preserving Attention Control (TAC) is used to maintain the fine details of clothing textures, thereby generating a realistic image corresponding to the input rendered image.