Identity‐consistent transfer learning of portraits for digital apparel sample display

Luyuan Wang,Yiqian Wu,Yong‐Liang Yang,Chen Liu,Xiaogang Jin
DOI: https://doi.org/10.1002/cav.2278
IF: 1.01
2024-06-11
Computer Animation and Virtual Worlds
Abstract:Our study presents a novel photo‐realistic portrait generation framework to mitigate the "uncanny valley" effect in digital apparel displays. By employing transfer learning, we map rendered avatars to realistic portraits while maintaining facial identity. Using the new DRFHQ dataset, we fine‐tuned the StyleGAN2‐FFHQ generator, achieving superior results in authenticity and user experience compared to existing methods. The rapid development of the online apparel shopping industry demands innovative solutions for high‐quality digital apparel sample displays with virtual avatars. However, developing such displays is prohibitively expensive and prone to the well‐known "uncanny valley" effect, where a nearly human‐looking artifact arouses eeriness and repulsiveness, thus affecting the user experience. To effectively mitigate the "uncanny valley" effect and improve the overall authenticity of digital apparel sample displays, we present a novel photo‐realistic portrait generation framework. Our key idea is to employ transfer learning to learn an identity‐consistent mapping from the latent space of rendered portraits to that of real portraits. During the inference stage, the input portrait of an avatar can be directly transferred to a realistic portrait by changing its appearance style while maintaining the facial identity. To this end, we collect a new dataset, Daz‐Rendered‐Faces‐HQ (DRFHQ), specifically designed for rendering‐style portraits. We leverage this dataset to fine‐tune the StyleGAN2‐FFHQ generator, using our carefully crafted framework, which helps to preserve the geometric and color features relevant to facial identity. We evaluate our framework using portraits with diverse gender, age, and race variations. Qualitative and quantitative evaluations, along with ablation studies, highlight our method's advantages over state‐of‐the‐art approaches.
computer science, software engineering
What problem does this paper attempt to address?