Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding

Zhimin Li,Jianwei Zhang,Qin Lin,Jiangfeng Xiong,Yanxin Long,Xinchi Deng,Yingfang Zhang,Xingchao Liu,Minbin Huang,Zedong Xiao,Dayou Chen,Jiajun He,Jiahao Li,Wenyue Li,Chen Zhang,Rongwei Quan,Jianxiang Lu,Jiabin Huang,Xiaoyan Yuan,Xiaoxiao Zheng,Yixuan Li,Jihong Zhang,Chao Zhang,Meng Chen,Jie Liu,Zheng Fang,Weiyan Wang,Jinbao Xue,Yangyu Tao,Jianchen Zhu,Kai Liu,Sihuan Lin,Yifu Sun,Yun Li,Dongdong Wang,Mingtao Chen,Zhichao Hu,Xiao Xiao,Yan Chen,Yuhong Liu,Wei Liu,Di Wang,Yong Yang,Jie Jiang,Qinglin Lu

2024-05-15

Abstract:We present Hunyuan-DiT, a text-to-image diffusion transformer with fine-grained understanding of both English and Chinese. To construct Hunyuan-DiT, we carefully design the transformer structure, text encoder, and positional encoding. We also build from scratch a whole data pipeline to update and evaluate data for iterative model optimization. For fine-grained language understanding, we train a Multimodal Large Language Model to refine the captions of the images. Finally, Hunyuan-DiT can perform multi-turn multimodal dialogue with users, generating and refining images according to the context. Through our holistic human evaluation protocol with more than 50 professional human evaluators, Hunyuan-DiT sets a new state-of-the-art in Chinese-to-image generation compared with other open-source models. Code and pretrained models are publicly available at

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper introduces a multi-resolution diffusive transformer called Hunyuan-DiT, which is capable of understanding and generating high-quality images in both English and Chinese. The main issue lies in the limited understanding of Chinese prompts by current text-to-image generation models, and Hunyuan-DiT addresses this problem through improved architecture and data processing flow. Specifically, it includes:

1. A novel network structure based on diffusive transformation, combining bilingual CLIP and multilingual T5 encoders to enhance language understanding.
2. A data processing pipeline built from scratch for model updating and optimization.
3. Utilization of a multimodal large-scale language model to improve original image captions, adding structural descriptions with world knowledge.
4. Support for multi-round multimodal dialogues with users, generating and modifying images based on context.
5. Post-training optimization during the inference stage to reduce deployment costs.

Hunyuan-DiT demonstrates outstanding performance in various aspects, such as text-image consistency, thematic clarity, and aesthetic effects. It outperforms other open-source models in Chinese-to-image generation and is capable of handling long text prompts, supporting image generation at multiple resolutions, and generating images in multi-round dialogues.