Abstract:The success of autoregressive (AR) language models in text generation has inspired the computer vision community to adopt Large Language Models (LLMs) for image generation. However, considering the essential differences between text and image modalities, the design space of language models for image generation remains underexplored. We observe that image tokens exhibit greater randomness compared to text tokens, which presents challenges when training with token prediction. Nevertheless, AR models demonstrate their potential by effectively learning patterns even from a seemingly suboptimal optimization problem. Our analysis also reveals that while all models successfully grasp the importance of local information in image generation, smaller models struggle to capture the global context. In contrast, larger models showcase improved capabilities in this area, helping to explain the performance gains achieved when scaling up model size. We further elucidate the design space of language models for vision generation, including tokenizer choice, model choice, model scalability, vocabulary design, and sampling strategy through extensive comparative experiments. Our work is the first to analyze the optimization behavior of language models in vision generation, and we believe it can inspire more effective designs when applying LMs to other domains. Finally, our elucidated language model for image generation, termed as ELM, achieves state-of-the-art performance on the ImageNet 256*256 benchmark. The code is available at <a class="link-external link-https" href="https://github.com/Pepperlll/LMforImageGeneration.git" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to explore and understand the design space of language models in image - generation tasks, especially how to apply large - language models (LLMs) to image generation. Specifically, the paper focuses on the following aspects: 1. **Modal Difference Analysis**: Although text and image are two different modalities, their discretized representations (i.e., token sequences) can be processed using similar methods. However, image tokens exhibit greater randomness, which makes training based on token prediction more challenging. The paper reveals the fundamental differences between image - token distributions and text - token distributions through quantitative analysis. 2. **Model Selection**: The paper compares two mainstream language - model methods - autoregressive models (AR) and masked language models (MLM), and finds that AR models perform better in image - generation tasks, especially in capturing local information and global context. 3. **Quantizer Selection**: The paper compares two common image - quantization methods - VQGAN and binary auto - encoder (BAE). The experimental results show that BAE is superior to VQGAN in code utilization, reconstruction ability, and generation performance. 4. **Vocabulary Design**: The paper explores the influence of different code dimensions on model performance and proposes an effective vocabulary - decomposition strategy, that is, splitting the original code into multiple sub - codes to reduce the learning complexity and improve performance. 5. **Sampling Strategy**: The paper studies different sampling strategies, including CFG scale, randomness introduction, and number of iterations, etc., to optimize the generation effect. 6. **ELM Model**: Based on the above research, the paper proposes a final model named ELM (Elucidated Language Model for Image Generation). ELM uses BAE as an image tokenizer and AR models for modeling. Through a series of optimization designs, ELM achieves state - of - the - art performance in the ImageNet 256×256 benchmark test. ### Formula Summary - **VQGAN Quantization Formula**: \[ z_q=\text{QUANT}(z): = \arg\min_{q_k\in Q}\|z_{ij}-q_k\|\in R^{H_f\times W_f\times D} \] - **BAE Quantization Formula**: \[ z^q_d = \text{sign}(z_d)= \begin{cases} 0 & \text{if }z_d < 0.5\\ 1 & \text{otherwise} \end{cases} \] - **AR Model Loss Function**: \[ L_{\text{ar}}=-E_{x\sim p(x)}[\log p(q)]=-E_{x\sim p(x)}\left[\sum_{l = 1}^{L}\log p(q_l)\right] \] - **MLM Model Loss Function**: \[ L_{\text{mlm}}=-E_{x\sim p(x)}\left[\sum_{\forall l\in[0, L], m_l = 1}\log p(q_l|q_M)\right] \] These formulas show the key technical details involved in the paper, helping readers better understand the working principle and optimization process of the model.

Elucidating the design space of language models for image generation

LLMGA: Multimodal Large Language Model based Generation Assistant

An Empirical Study and Analysis of Text-to-Image Generation Using Large Language Model-Powered Textual Representation

Leveraging Large Language Models for Scalable Vector Graphics-Driven Image Understanding

Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation

Beyond Text: Frozen Large Language Models in Visual Signal Comprehension

VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks

Planting a SEED of Vision in Large Language Model

Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective

Leopard: A Vision Language Model For Text-Rich Multi-Image Tasks

Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models

Rethinking VLMs and LLMs for Image Classification

Making LLaMA SEE and Draw with SEED Tokenizer

LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models

Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions

LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation

Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation

Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

Pixel Aligned Language Models

Multi-modal Auto-regressive Modeling via Visual Words

Improving Visual Commonsense in Language Models via Multiple Image Generation