Abstract:Text-to-image synthesis aims to generate a photo-realistic and semantic consistent image from a specific text description. The images synthesized by off-the-shelf models usually contain limited components compared with the corresponding image and text description, which decreases the image quality and the textual-visual consistency. To address this issue, we propose a novel Vision-Language Matching strategy for text-to-image synthesis, named VLMGAN*, which introduces a dual vision-language matching mechanism to strengthen the image quality and semantic consistency. The dual vision-language matching mechanism considers textual-visual matching between the generated image and the corresponding text description, and visual-visual consistent constraints between the synthesized image and the real image. Given a specific text description, VLMGAN* firstly encodes it into textual features and then feeds them to a dual vision-language matching-based generative model to synthesize a photo-realistic and textual semantic consistent image. Besides, the popular evaluation metrics for text-to-image synthesis are borrowed from simple image generation, which mainly evaluates the reality and diversity of the synthesized images. Therefore, we introduce a metric named Vision-Language Matching Score (VLMS) to evaluate the performance of text-to-image synthesis which can consider both the image quality and the semantic consistency between synthesized image and the description. The proposed dual multi-level vision-language matching strategy can be applied to other text-to-image synthesis methods. We implement this strategy on two popular baselines, which are marked with ${\text{VLMGAN}_{+\text{AttnGAN}}}$ and ${\text{VLMGAN}_{+\text{DFGAN}}}$. The experimental results on two widely-used datasets show that the model achieves significant improvements over other state-of-the-art methods.

Unsupervised Text-to-image Synthesis

Learn, Imagine and Create: Text-to-Image Generation from Prior Knowledge.

Unsupervised Image-to-Image Translation with Generative Adversarial Networks.

Text-to-image Synthesis via Symmetrical Distillation Networks

Multi-Semantic Fusion Generative Adversarial Network for Text-to-Image Generation

Semantic Image Synthesis via Adversarial Learning

Semantics Disentangling for Text-to-Image Generation

SAM-GAN: Self-Attention supporting Multi-stage Generative Adversarial Networks for text-to-image synthesis

Disentangling for Text-to-Image Generation

Perceptual Pyramid Adversarial Networks for Text-to-Image Synthesis.

Multi-Sentence Auxiliary Adversarial Networks for Fine-Grained Text-to-Image Synthesis

Vision-Language Matching for Text-to-Image Synthesis via Generative Adversarial Networks

Training-Free Location-Aware Text-to-Image Synthesis

Object-driven Text-to-Image Synthesis via Adversarial Training

Verisimilar Image Synthesis for Accurate Detection and Recognition of Texts in Scenes

Zero-shot Text-guided Infinite Image Synthesis with LLM guidance

Training-Free Consistent Text-to-Image Generation

Label-free Neural Semantic Image Synthesis

Learning to Draw Text in Natural Images with Conditional Adversarial Networks

Text-Free Image-to-Speech Synthesis Using Learned Segmental Units