Arbitrary Style Guidance for Enhanced Diffusion-Based Text-to-Image Generation

Zhihong Pan,Xin Zhou,Hao Tian
DOI: https://doi.org/10.48550/arXiv.2211.07751
2022-11-15
Abstract:Diffusion-based text-to-image generation models like GLIDE and DALLE-2 have gained wide success recently for their superior performance in turning complex text inputs into images of high quality and wide diversity. In particular, they are proven to be very powerful in creating graphic arts of various formats and styles. Although current models supported specifying style formats like oil painting or pencil drawing, fine-grained style features like color distributions and brush strokes are hard to specify as they are randomly picked from a conditional distribution based on the given text input. Here we propose a novel style guidance method to support generating images using arbitrary style guided by a reference image. The generation method does not require a separate style transfer model to generate desired styles while maintaining image quality in generated content as controlled by the text input. Additionally, the guidance method can be applied without a style reference, denoted as self style guidance, to generate images of more diverse styles. Comprehensive experiments prove that the proposed method remains robust and effective in a wide range of conditions, including diverse graphic art forms, image content types and diffusion models.
Computer Vision and Pattern Recognition,Image and Video Processing
What problem does this paper attempt to address?
The problem that this paper attempts to solve is how to more finely control the artistic style of the generated images in text - to - image generation based on diffusion models. Although existing models such as GLIDE and DALLE - 2 are already able to generate high - quality and diverse images according to complex text inputs, there are still limitations in specifying fine - grained style features (such as color distribution and brushstrokes). These features are usually randomly selected from the conditional distribution of the given text input and are difficult to specify precisely. To solve this problem, the author proposes a new style - guiding method that allows the use of reference images to guide the generation of images with arbitrary styles. This method does not require an additional style - transfer model and can generate images in the desired style while maintaining the content quality controlled by the text input. In addition, a self - style - guiding method is also proposed, which can generate more diverse style images without style references. Through extensive experiments, it is proved that the proposed method can maintain robustness and effectiveness under a variety of conditions, including different graphic art forms, image content types and diffusion models.