Abstract:Text-to-image (T2I) generation models have significantly advanced in recent years. However, effective interaction with these models is challenging for average users due to the need for specialized prompt engineering knowledge and the inability to perform multi-turn image generation, hindering a dynamic and iterative creation process. Recent attempts have tried to equip Multi-modal Large Language Models (MLLMs) with T2I models to bring the user's natural language instructions into reality. Hence, the output modality of MLLMs is extended, and the multi-turn generation quality of T2I models is enhanced thanks to the strong multi-modal comprehension ability of MLLMs. However, many of these works face challenges in identifying correct output modalities and generating coherent images accordingly as the number of output modalities increases and the conversations go deeper. Therefore, we propose DialogGen, an effective pipeline to align off-the-shelf MLLMs and T2I models to build a Multi-modal Interactive Dialogue System (MIDS) for multi-turn Text-to-Image generation. It is composed of drawing prompt alignment, careful training data curation, and error correction. Moreover, as the field of MIDS flourishes, comprehensive benchmarks are urgently needed to evaluate MIDS fairly in terms of output modality correctness and multi-modal output coherence. To address this issue, we introduce the Multi-modal Dialogue Benchmark (DialogBen), a comprehensive bilingual benchmark designed to assess the ability of MLLMs to generate accurate and coherent multi-modal content that supports image editing. It contains two evaluation metrics to measure the model's ability to switch modalities and the coherence of the output images. Our extensive experiments on DialogBen and user study demonstrate the effectiveness of DialogGen compared with other State-of-the-Art models.

Teaching Text-to-Image Models to Communicate.

Teaching Text-to-Image Models to Communicate in Dialog

Teaching Machines to Converse

An End-to-End Model for Photo-Sharing Multi-modal Dialogue Generation

Reflective Human-Machine Co-adaptation for Enhanced Text-to-Image Generation Dialogue System

ArtAug: Enhancing Text-to-Image Generation through Synthesis-Understanding Interaction

Customization Assistant for Text-to-image Generation

Emotional Dialogue Generation using Image-Grounded Language Models

DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generation

ChatGen: Automatic Text-to-Image Generation From FreeStyle Chatting

ChatPainter: Improving Text to Image Generation using Dialogue

Visualizing Dialogues: Enhancing Image Selection through Dialogue Understanding with Large Language Models

Mini-DALLE3: Interactive Text to Image by Prompting Large Language Models

Open Domain Dialogue Generation with Latent Images

DiffChat: Learning to Chat with Text-to-Image Synthesis Models for Interactive Image Creation

AutoStudio: Crafting Consistent Subjects in Multi-turn Interactive Image Generation

Customizing Text-to-Image Generation with Inverted Interaction

Dialog-based Interactive Image Retrieval

Exploring Contextual-Aware Representation and Linguistic-Diverse Expression for Visual Dialog.

Maria: A Visual Experience Powered Conversational Agent

Unified Text-to-Image Generation and Retrieval