Language-Driven Image Style Transfer

Tsu-Jui Fu,Xin Eric Wang,William Yang Wang
DOI: https://doi.org/10.48550/arXiv.2106.00178
2022-07-18
Abstract:Despite having promising results, style transfer, which requires preparing style images in advance, may result in lack of creativity and accessibility. Following human instruction, on the other hand, is the most natural way to perform artistic style transfer that can significantly improve controllability for visual effect applications. We introduce a new task, language-driven artistic style transfer (LDAST), to manipulate the style of a content image, guided by a text. We propose contrastive language visual artist (CLVA) that learns to extract visual semantics from style instructions and accomplish LDAST by the patch-wise style discriminator. The discriminator considers the correlation between language and patches of style images or transferred results to jointly embed style instructions. CLVA further compares contrastive pairs of content images and style instructions to improve the mutual relativeness. The results from the same content image can preserve consistent content structures. Besides, they should present analogous style patterns from style instructions that contain similar visual semantics. The experiments show that our CLVA is effective and achieves superb transferred results on LDAST.
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the lack of creativity and ease - of - use in artistic style transfer. Traditional style transfer methods require pre - prepared style images, which not only limit the exertion of creativity but also reduce the usability of the methods. The paper proposes a new task - Language - driven Artistic Style Transfer (LDAST), which uses text instructions to guide the style transfer process, thereby enhancing the controllability and creativity in visual effects applications. Specifically, the paper introduces a model named Contrastive Language Visual Artist (CLVA). This model can extract visual semantics from style instructions and implement LDAST through a patch - wise style discriminator. CLVA also compares contrast pairs between different content images and style instructions through contrastive reasoning to strengthen the mutual correlation. Experimental results show that CLVA performs well in both automatic metrics and human evaluations and can effectively achieve language - driven artistic style transfer.