Abstract:We propose a method to fuse frozen text-only large language models (LLMs) with pre-trained image encoder and decoder models, by mapping between their embedding spaces. Our model demonstrates a wide suite of multimodal capabilities: image retrieval, novel image generation, and multimodal dialogue. Ours is the first approach capable of conditioning on arbitrarily interleaved image and text inputs to generate coherent image (and text) outputs. To achieve strong performance on image generation, we propose an efficient mapping network to ground the LLM to an off-the-shelf text-to-image generation model. This mapping network translates hidden representations of text into the embedding space of the visual models, enabling us to leverage the strong text representations of the LLM for visual outputs. Our approach outperforms baseline generation models on tasks with longer and more complex language. In addition to novel image generation, our model is also capable of image retrieval from a prespecified dataset, and decides whether to retrieve or generate at inference time. This is done with a learnt decision module which conditions on the hidden representations of the LLM. Our model exhibits a wider range of capabilities compared to prior multimodal language models. It can process image-and-text inputs, and produce retrieved images, generated images, and generated text -- outperforming non-LLM based generation models across several text-to-image tasks that measure context dependence.

What problem does this paper attempt to address?

The paper attempts to address the problem of how to integrate large language models (LLMs) with pre-trained image encoder and decoder models to achieve multimodal tasks such as image retrieval, new image generation, and multimodal dialogue. Specifically, the authors propose a method that establishes a mapping of the embedding space between the frozen text language model and the image generation model, enabling the model to handle arbitrarily interleaved image and text inputs and generate coherent image or text outputs. ### Main Issues 1. **Multimodal Capability Expansion**: - How to enable large language models to have multimodal capabilities, i.e., to handle image and text inputs and generate image or text outputs. - How to achieve this goal using existing large language models and image generation models without retraining the entire model. 2. **Image Generation and Retrieval**: - How to generate high-quality new images given a text description. - How to retrieve the most relevant images from a predefined dataset given a text description. - How to decide between generating an image or retrieving an image. 3. **Handling Long Texts and Complex Contexts**: - How to improve the model's performance in handling long texts and complex contexts, especially in dialogues and narrative texts. ### Solution The authors propose a model named GILL (Generating Images with Large Language Models) that addresses the above issues through the following methods: 1. **Embedding Space Mapping**: - By using a small mapping network (GILLMapper), the hidden representations of the text language model are converted into the embedding space of the image generation model, enabling image generation. - This mapping network is trained by minimizing the L2 distance between its output and the output of the image generation model's text encoder. 2. **Multimodal Input Handling**: - The model can handle arbitrarily interleaved image and text inputs and generate corresponding image or text outputs. - By introducing a special [IMG] token, the model can decide when to generate an image or text during the generation process. 3. **Image Generation and Retrieval Decision**: - Through a decision module, the model can choose between generating an image and retrieving an image. This decision module is trained based on the hidden representations of the language model. 4. **Experimental Validation**: - Experiments were conducted on multiple datasets, including VIST and VisDial, validating that GILL outperforms baseline models in handling long texts and complex contexts. - Experimental results show that GILL excels in both image generation and retrieval tasks, especially when handling multimodal inputs. ### Summary By proposing the GILL model, the paper successfully combines large language models with image generation models, achieving the expansion of multimodal task capabilities. GILL not only generates high-quality new images but also retrieves relevant images from predefined datasets and makes intelligent decisions between generation and retrieval. These features make GILL perform excellently in handling long texts and complex contexts, providing a new solution for multimodal tasks.

Generating Images with Multimodal Language Models

Multimodal Latent Language Modeling with Next-Token Diffusion

Grounding Language Models to Images for Multimodal Inputs and Outputs

Unified Generative and Discriminative Training for Multi-modal Large Language Models

LLMGA: Multimodal Large Language Model based Generation Assistant

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

Improving Visual Commonsense in Language Models via Multiple Image Generation

InfMLLM: A Unified Framework for Visual-Language Tasks.

Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation

An Empirical Study and Analysis of Text-to-Image Generation Using Large Language Model-Powered Textual Representation

VLIS: Unimodal Language Models Guide Multimodal Language Generation

Retrieval-Augmented Multimodal Language Modeling

Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond

Synth$^2$: Boosting Visual-Language Models with Synthetic Captions and Image Embeddings

Improving Context Understanding in Multimodal Large Language Models Via Multimodal Composition Learning

MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

Liquid: Language Models are Scalable Multi-modal Generators

MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer

Linearly Mapping from Image to Text Space

EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model

UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion