FonTS: Text Rendering with Typography and Style Controls

Wenda Shi,Yiren Song,Dengming Zhang,Jiaming Liu,Xingxing Zou

2024-11-29

Abstract:Visual text images are prevalent in various applications, requiring careful font selection and typographic choices. Recent advances in Diffusion Transformer (DiT)-based text-to-image (T2I) models show promise in automating these processes. However, these methods still face challenges such as inconsistent fonts, style variation, and limited fine-grained control, particularly at the word level. This paper proposes a two-stage DiT-based pipeline to address these issues by enhancing controllability over typography and style in text rendering. We introduce Typography Control (TC) finetuning, an efficient parameter fine-tuning method, and enclosing typography control tokens (ETC-tokens), which enable precise word-level application of typographic features. To further enhance style control, we present a Style Control Adapter (SCA) that injects style information through image inputs independent of text prompts. Through comprehensive experiments, we demonstrate the effectiveness of our approach in achieving superior word-level typographic control, font consistency, and style consistency in Basic and Artistic Text Rendering (BTR and ATR) tasks. Our results mark a significant advancement in the precision and adaptability of T2I models, presenting new possibilities for creative applications and design-oriented tasks.

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The problems that this paper attempts to solve mainly include:

1. **Font and Word - level Attribute Control**: In basic text rendering (BTR), although existing methods can generate high - quality text images, they are insufficient in font selection and word - level style control. For example, it is difficult to precisely apply styles such as bold, italic, or underline to specific words.

2. **Style Consistency**: In artistic text rendering (ATR), when existing methods generate artistic texts with specific styles, the problem of style inconsistency often occurs. This limits the application of artistic texts in the design and creative fields.

3. **Maintenance of Scene Text Rendering Ability**: While enhancing the controllability of text rendering, maintain the ability of scene text rendering (STR) and avoid having a negative impact on the performance of existing models.

To address these challenges, the paper proposes a two - stage pipeline based on the diffusion transformer (DiT). By introducing typography control (TC) fine - tuning and style control adapter (SCA), fine - grained control of fonts, word - level attributes, and styles in text rendering is achieved. Specifically:

- **Typography Control (TC)**: By introducing enclosing typography control tokens (ETC - tokens) and paired typography control datasets (TC - dataset), the model can precisely apply typography features such as bold, italic, and underline at specific word positions.

- **Style Control Adapter (SCA)**: Inject style information independently of text prompts through image input to enhance the flexibility and consistency of style control. SCA can handle various style conditions, including artistic text images and arbitrary style images, by being trained on large - scale image - text pair datasets.

The paper verifies the effectiveness of the proposed method through extensive experiments and shows superior performance in basic text rendering and artistic text rendering tasks, especially significant improvements in word - level typography control, font consistency, and style consistency. These improvements provide new possibilities for creative applications and design - oriented tasks.