ControlAR: Controllable Image Generation with Autoregressive Models

Zongming Li,Tianheng Cheng,Shoufa Chen,Peize Sun,Haocheng Shen,Longjin Ran,Xiaoxin Chen,Wenyu Liu,Xinggang Wang

2024-10-04

Abstract:Autoregressive (AR) models have reformulated image generation as next-token prediction, demonstrating remarkable potential and emerging as strong competitors to diffusion models. However, control-to-image generation, akin to ControlNet, remains largely unexplored within AR models. Although a natural approach, inspired by advancements in Large Language Models, is to tokenize control images into tokens and prefill them into the autoregressive model before decoding image tokens, it still falls short in generation quality compared to ControlNet and suffers from inefficiency. To this end, we introduce ControlAR, an efficient and effective framework for integrating spatial controls into autoregressive image generation models. Firstly, we explore control encoding for AR models and propose a lightweight control encoder to transform spatial inputs (e.g., canny edges or depth maps) into control tokens. Then ControlAR exploits the conditional decoding method to generate the next image token conditioned on the per-token fusion between control and image tokens, similar to positional encodings. Compared to prefilling tokens, using conditional decoding significantly strengthens the control capability of AR models but also maintains the model's efficiency. Furthermore, the proposed ControlAR surprisingly empowers AR models with arbitrary-resolution image generation via conditional decoding and specific controls. Extensive experiments can demonstrate the controllability of the proposed ControlAR for the autoregressive control-to-image generation across diverse inputs, including edges, depths, and segmentation masks. Furthermore, both quantitative and qualitative results indicate that ControlAR surpasses previous state-of-the-art controllable diffusion models, e.g., ControlNet++. Code, models, and demo will soon be available at <a class="link-external link-https" href="https://github.com/hustvl/ControlAR" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to address the problem of achieving controllable image generation in Autoregressive (AR) models. Specifically, while existing autoregressive models have shown significant potential in image generation, they still lack controllability compared to Diffusion Models. The goal of the paper is to enhance the controllability of autoregressive models by introducing a new framework—ControlAR, enabling them to generate high-quality images and support image generation at any resolution.

### Main Issues:
1. **How to encode 2D spatial control images**: Autoregressive models use sequence modeling and next-token prediction for image generation, so it is necessary to convert 2D spatial control images into a sequence form suitable for autoregressive models.
2. **How to use the encoded control information to guide image generation**: A method needs to be designed so that the autoregressive model can effectively use the control information when generating each image token.

### Solutions:
1. **Control Encoder**: The paper proposes a lightweight control encoder that converts 2D spatial control images (such as Canny edges, depth maps, etc.) into control token sequences.
2. **Conditional Decoding**: Unlike traditional prefilling methods, the paper proposes a conditional decoding method, which fuses the current control token with the previous image token to predict the next image token when generating each image token. This method not only improves the generation quality but also maintains the efficiency of the model.

### Main Contributions:
1. **Proposing the ControlAR framework**: This framework enables precise controllable image generation and produces high-quality images.
2. **Extending the control capabilities of autoregressive models**: Under different control conditions, ControlAR demonstrates performance comparable to or even better than state-of-the-art diffusion models.
3. **Supporting image generation at any resolution**: By adjusting the size of the input control image, ControlAR can generate images at any resolution without relying on prompts of specific resolutions.

### Experimental Results:
- **C2I (Class-to-Image) Controllable Generation**: On the ImageNet dataset, ControlAR achieved lower FID values on multiple control tasks (such as Canny edges, depth maps), indicating higher quality of generated images.
- **T2I (Text-to-Image) Controllable Generation**: On multiple datasets (such as ADE20K, COCOStuff, MultiGen-20M), ControlAR performed excellently on various control tasks (such as segmentation masks, Canny edges, HED edges, line art edges, depth maps), especially in terms of conditional consistency (such as mIoU, F1-Score, SSIM, RMSE) and the quality of generated images (FID).

In summary, by proposing the ControlAR framework, this paper successfully addresses the shortcomings of autoregressive models in controllable image generation, providing new possibilities for the application of autoregressive models in the field of image generation.