Fine-grained Text Style Transfer with Diffusion-Based Language Models

Yiwei Lyu,Tiange Luo,Jiacheng Shi,Todd C. Hollon,Honglak Lee

2023-06-12

Abstract:Diffusion probabilistic models have shown great success in generating high-quality images controllably, and researchers have tried to utilize this controllability into text generation domain. Previous works on diffusion-based language models have shown that they can be trained without external knowledge (such as pre-trained weights) and still achieve stable performance and controllability. In this paper, we trained a diffusion-based model on StylePTB dataset, the standard benchmark for fine-grained text style transfers. The tasks in StylePTB requires much more refined control over the output text compared to tasks evaluated in previous works, and our model was able to achieve state-of-the-art performance on StylePTB on both individual and compositional transfers. Moreover, our model, trained on limited data from StylePTB without external knowledge, outperforms previous works that utilized pretrained weights, embeddings, and external grammar parsers, and this may indicate that diffusion-based language models have great potential under low-resource settings.

Computation and Language,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The paper aims to address the problem of fine-grained text style transfer. Specifically, the researchers leverage the potential of diffusion models in the field of controllable text generation, particularly for tasks that require highly precise control over the output text. The model in the paper is trained on the StylePTB dataset, which is the standard benchmark for fine-grained text style transfer. Compared to previous models, this model achieves state-of-the-art performance in both individual and combined transfer tasks. Notably, this model surpasses those that utilize pre-trained weights and other tools, despite only using limited data from the StylePTB dataset and without relying on external knowledge (such as pre-trained weights, embeddings, or grammar parsers). This indicates that diffusion model-based language models have great potential in resource-constrained scenarios. Additionally, the study demonstrates the model's ability to perform multiple fine-grained transfers on a single sentence.

Fine-grained Text Style Transfer with Diffusion-Based Language Models

UATST: Towards Unpaired Arbitrary Text-Guided Style Transfer with Cross-Space Modulation

ControlStyle: Text-Driven Stylized Image Generation Using Diffusion Priors

FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Models

DiffuseST: Unleashing the Capability of the Diffusion Model for Style Transfer

Improving Diffusion Models for Scene Text Editing with Dual Encoders

Style Injection in Diffusion: A Training-Free Approach for Adapting Large-Scale Diffusion Models for Style Transfer

3Dstyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Models

Style-A-Video: Agile Diffusion for Arbitrary Text-Based Video Style Transfer

Memory-enhanced text style transfer with dynamic style learning and calibration

Arbitrary Style Guidance for Enhanced Diffusion-Based Text-to-Image Generation

Interpretable Style Transfer for Text-to-Speech with ControlVAE and Diffusion Bridge

DiffStyleTTS: Diffusion-based Hierarchical Prosody Modeling for Text-to-Speech with Diverse and Controllable Styles

ITstyler: Image-optimized Text-based Style Transfer

StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Models

Text-driven Visual Synthesis with Latent Diffusion Prior

Latent representation discretization for unsupervised text style generation

Reinforcement Learning Based Text Style Transfer without Parallel Training Corpus

Style Transfer with Multi-iteration Preference Optimization

ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Models