Abstract:Despite significant progress in generative AI, comprehensive evaluation remains challenging because of the lack of effective metrics and standardized benchmarks. For instance, the widely-used CLIPScore measures the alignment between a (generated) image and text prompt, but it fails to produce reliable scores for complex prompts involving compositions of objects, attributes, and relations. One reason is that text encoders of CLIP can notoriously act as a "bag of words", conflating prompts such as "the horse is eating the grass" with "the grass is eating the horse". To address this, we introduce the VQAScore, which uses a visual-question-answering (VQA) model to produce an alignment score by computing the probability of a "Yes" answer to a simple "Does this figure show '{text}'?" question. Though simpler than prior art, VQAScore computed with off-the-shelf models produces state-of-the-art results across many (8) image-text alignment benchmarks. We also compute VQAScore with an in-house model that follows best practices in the literature. For example, we use a bidirectional image-question encoder that allows image embeddings to depend on the question being asked (and vice versa). Our in-house model, CLIP-FlanT5, outperforms even the strongest baselines that make use of the proprietary GPT-4V. Interestingly, although we train with only images, VQAScore can also align text with video and 3D models. VQAScore allows researchers to benchmark text-to-visual generation using complex texts that capture the compositional structure of real-world prompts. We introduce GenAI-Bench, a more challenging benchmark with 1,600 compositional text prompts that require parsing scenes, objects, attributes, relationships, and high-order reasoning like comparison and logic. GenAI-Bench also offers over 15,000 human ratings for leading image and video generation models such as Stable Diffusion, DALL-E 3, and Gen2.

VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance

Learn, Imagine and Create: Text-to-Image Generation from Prior Knowledge.

CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP

Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts

CgT-GAN: CLIP-guided Text GAN for Image Captioning

Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation

TextCLIP: Text-Guided Face Image Generation And Manipulation Without Adversarial Training

Evaluating Text-to-Visual Generation with Image-to-Text Generation

DeltaSpace: A Semantic-aligned Feature Space for Flexible Text-guided Image Editing

CLIP-VQDiffusion : Langauge Free Training of Text To Image generation using CLIP and vector quantized diffusion model

In the Era of Prompt Learning with Vision-Language Models

GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesis

Unknown Prompt, the only Lacuna: Unveiling CLIP's Potential for Open Domain Generalization

CLIP-Guided StyleGAN Inversion for Text-Driven Real Image Editing

How Much Can CLIP Benefit Vision-and-Language Tasks?

GLIGEN: Open-Set Grounded Text-to-Image Generation

CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Grounding

CLIP Model for Images to Textual Prompts Based on Top-k Neighbors

Controllable Image Generation With Composed Parallel Token Prediction

DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation

Distilling Knowledge from Text-to-Image Generative Models Improves Visio-Linguistic Reasoning in CLIP