Abstract:In many text-generation problems, users may prefer not only a single response, but a diverse range of high-quality outputs from which to choose. Quality-diversity (QD) search algorithms aim at such outcomes, by continually improving and diversifying a population of candidates. However, the applicability of QD to qualitative domains, like creative writing, has been limited by the difficulty of algorithmically specifying measures of quality and diversity. Interestingly, recent developments in language models (LMs) have enabled guiding search through AI feedback, wherein LMs are prompted in natural language to evaluate qualitative aspects of text. Leveraging this development, we introduce Quality-Diversity through AI Feedback (QDAIF), wherein an evolutionary algorithm applies LMs to both generate variation and evaluate the quality and diversity of candidate text. When assessed on creative writing domains, QDAIF covers more of a specified search space with high-quality samples than do non-QD controls. Further, human evaluation of QDAIF-generated creative texts validates reasonable agreement between AI and human evaluation. Our results thus highlight the potential of AI feedback to guide open-ended search for creative and original solutions, providing a recipe that seemingly generalizes to many domains and modalities. In this way, QDAIF is a step towards AI systems that can independently search, diversify, evaluate, and improve, which are among the core skills underlying human society's capacity for innovation.

Informed Sampling for Diversity in Concept-to-Text NLG

A Simple, Fast Diverse Decoding Algorithm for Neural Generation

Large Language Models as In-context AI Generators for Quality-Diversity

Generalising Multilingual Concept-to-Text NLG with Language Agnostic Delexicalisation

IFDID: Information Filter upon Diversity-Improved Decoding for Diversity-Faithfulness Tradeoff in NLG

Improving Diversity of Commonsense Generation by Large Language Models via In-Context Learning

Improving Diversity of Neural Text Generation Via Inverse Probability Weighting

Semantic Diversity in Dialogue with Natural Language Inference

Screening Through a Broad Pool: Towards Better Diversity for Lexically Constrained Text Generation

Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentation

Diversity-Promoting GAN: A Cross-Entropy Based Generative Adversarial Network for Diversified Text Generation

The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text

Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions

Evaluating Diversity in Automatic Poetry Generation

Diversifying Neural Text Generation with Part-of-Speech Guided Softmax and Sampling

A Learning-Exploring Method to Generate Diverse Paraphrases with Multi-Objective Deep Reinforcement Learning

DP-GAN: Diversity-Promoting Generative Adversarial Network for Generating Informative and Diversified Text

Quality-Diversity through AI Feedback

Controllable Text Generation for Open-Domain Creativity and Fairness

Improve the Diversity and Novelty for Open-Ended Neural Text Generation via Inverse Probability Weighting.

Growing a Tail: Increasing Output Diversity in Large Language Models