Abstract:Diffusion model based Text-to-Image has achieved impressive achievements recently. Although current technology for synthesizing images is highly advanced and capable of generating images with high fidelity, it is still possible to give the show away when focusing on the text area in the generated image. To address this issue, we introduce AnyText, a diffusion-based multilingual visual text generation and editing model, that focuses on rendering accurate and coherent text in the image. AnyText comprises a diffusion pipeline with two primary elements: an auxiliary latent module and a text embedding module. The former uses inputs like text glyph, position, and masked image to generate latent features for text generation or editing. The latter employs an OCR model for encoding stroke data as embeddings, which blend with image caption embeddings from the tokenizer to generate texts that seamlessly integrate with the background. We employed text-control diffusion loss and text perceptual loss for training to further enhance writing accuracy. AnyText can write characters in multiple languages, to the best of our knowledge, this is the first work to address multilingual visual text generation. It is worth mentioning that AnyText can be plugged into existing diffusion models from the community for rendering or editing text accurately. After conducting extensive evaluation experiments, our method has outperformed all other approaches by a significant margin. Additionally, we contribute the first large-scale multilingual text images dataset, AnyWord-3M, containing 3 million image-text pairs with OCR annotations in multiple languages. Based on AnyWord-3M dataset, we propose AnyText-benchmark for the evaluation of visual text generation accuracy and quality. Our project will be open-sourced on <a class="link-external link-https" href="https://github.com/tyxsspa/AnyText" rel="external noopener nofollow">this https URL</a> to improve and promote the development of text generation technology.

Emage: Non-Autoregressive Text-to-Image Generation

Diversified text-to-image generation via deep mutual information estimation

RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Model

Implementing and Experimenting with Diffusion Models for Text-to-Image Generation

Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models

Vector Quantized Diffusion Model for Text-to-Image Synthesis

SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Seconds

Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers

Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesis

Diffusion Beats Autoregressive: An Evaluation of Compositional Generation in Text-to-Image Models

TextCraftor: Your Text Encoder Can be Image Quality Controller

ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Models

Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference

LaDiC: Are Diffusion Models Really Inferior to Autoregressive Counterparts for Image-to-Text Generation?

AnyText: Multilingual Visual Text Generation And Editing

RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths

MobileDiffusion: Instant Text-to-Image Generation on Mobile Devices

Rapid Diffusion: Building Domain-Specific Text-to-Image Synthesizers with Fast Inference Speed

I4VGen: Image as Free Stepping Stone for Text-to-Video Generation