Language-Guided Image Tokenization for Generation

Kaiwen Zha,Lijun Yu,Alireza Fathi,David A. Ross,Cordelia Schmid,Dina Katabi,Xiuye Gu

2024-12-08

Abstract:Image tokenization, the process of transforming raw image pixels into a compact low-dimensional latent representation, has proven crucial for scalable and efficient image generation. However, mainstream image tokenization methods generally have limited compression rates, making high-resolution image generation computationally expensive. To address this challenge, we propose to leverage language for efficient image tokenization, and we call our method Text-Conditioned Image Tokenization (TexTok). TexTok is a simple yet effective tokenization framework that leverages language to provide high-level semantics. By conditioning the tokenization process on descriptive text captions, TexTok allows the tokenization process to focus on encoding fine-grained visual details into latent tokens, leading to enhanced reconstruction quality and higher compression rates. Compared to the conventional tokenizer without text conditioning, TexTok achieves average reconstruction FID improvements of 29.2% and 48.1% on ImageNet-256 and -512 benchmarks respectively, across varying numbers of tokens. These tokenization improvements consistently translate to 16.3% and 34.3% average improvements in generation FID. By simply replacing the tokenizer in Diffusion Transformer (DiT) with TexTok, our system can achieve a 93.5x inference speedup while still outperforming the original DiT using only 32 tokens on ImageNet-512. TexTok with a vanilla DiT generator achieves state-of-the-art FID scores of 1.46 and 1.62 on ImageNet-256 and -512 respectively. Furthermore, we demonstrate TexTok's superiority on the text-to-image generation task, effectively utilizing the off-the-shelf text captions in tokenization.

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to solve the problem of efficient image tokenization in image generation, especially in high - resolution image generation, where traditional methods have a trade - off between compression rate and reconstruction quality. Specifically:

1. **Trade - off between compression rate and reconstruction quality**: Existing image tokenization methods usually face a dilemma in high - resolution image generation: a high compression rate reduces the computational cost but often sacrifices the reconstruction quality; giving priority to quality will lead to extremely high computational overhead.

2. **Introduction of text - guided image tokenization**: To solve the above problems, the author proposes a new method - **Text - Conditioned Image Tokenization (TexTok)**. By using descriptive text captions, TexTok can provide high - level semantic information during the tokenization process, enabling the tokenizer to focus more on encoding fine - grained visual details, thereby improving the reconstruction quality and compression rate.

3. **Improvement of generation performance**: In addition to improving the reconstruction quality, TexTok also significantly improves the quality of image generation. By applying TexTok to the Diffusion Transformer (DiT), the system not only improves the inference speed by 93.5 times but also outperforms the original DiT on ImageNet - 512 with only 32 tokens.

4. **Multi - task applicability**: In addition, the author also demonstrates the superiority of TexTok in text - to - image generation tasks, effectively using off - the - shelf text captions for tokenization, further improving the quality of the generated images.

In summary, this paper solves the trade - off problem between the compression rate and reconstruction quality of existing tokenization methods in high - resolution image generation by introducing the text - guided image tokenization method and demonstrates its superior performance in multiple tasks.