Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Support

Xiaojun Wu,Dixiang Zhang,Ruyi Gan,Junyu Lu,Ziwei Wu,Renliang Sun,Jiaxing Zhang,Pingjian Zhang,Yan Song

2024-06-18

Abstract:Recent advancements in text-to-image models have significantly enhanced image generation capabilities, yet a notable gap of open-source models persists in bilingual or Chinese language support. To address this need, we present Taiyi-Diffusion-XL, a new Chinese and English bilingual text-to-image model which is developed by extending the capabilities of CLIP and Stable-Diffusion-XL through a process of bilingual continuous pre-training. This approach includes the efficient expansion of vocabulary by integrating the most frequently used Chinese characters into CLIP's tokenizer and embedding layers, coupled with an absolute position encoding expansion. Additionally, we enrich text prompts by large vision-language model, leading to better images captions and possess higher visual quality. These enhancements are subsequently applied to downstream text-to-image models. Our empirical results indicate that the developed CLIP model excels in bilingual image-text retrieval.Furthermore, the bilingual image generation capabilities of Taiyi-Diffusion-XL surpass previous models. This research leads to the development and open-sourcing of the Taiyi-Diffusion-XL model, representing a notable advancement in the field of image generation, particularly for Chinese language applications. This contribution is a step forward in addressing the need for more diverse language support in multimodal research. The model and demonstration are made publicly available at \href{<a class="link-external link-https" href="https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/" rel="external noopener nofollow">this https URL</a>}, fostering further research and collaboration in this domain.

Computation and Language

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper mainly addresses the following issues:

1. **Lack of bilingual text-to-image generation models**:
Most current open-source text-to-image models only support English, and very few models can support both Chinese and English. To address this gap, the paper proposes Taiyi-Diffusion-XL (Taiyi-XL), a text-to-image generation model specifically optimized for bilingual Chinese and English scenarios.

2. **Vocabulary expansion and positional encoding**:
Enhancing the model's performance in a bilingual environment by expanding the vocabulary and positional encoding. Specifically, integrating commonly used Chinese characters into the CLIP tokenizer and embedding layer, and extending absolute positional encoding to improve the model's understanding of the text.

3. **Enriching text prompts using large-scale vision-language models**:
Using large-scale vision-language models to enrich text prompts, resulting in generated images with higher visual quality and more detailed descriptive information.

4. **Development and open-sourcing of the bilingual model**:
Developing and open-sourcing the Taiyi-Diffusion-XL model to promote the advancement of bilingual text-to-image generation technology, particularly in Chinese applications.

In summary, the paper aims to enhance the performance of bilingual text-to-image generation models, especially in supporting Chinese, to meet the needs of multilingual application scenarios.