Abstract:Text detoxification, a variant of style transfer tasks, finds useful applications in online social media. This work presents a fine-tuning method that only uses non-parallel data to turn large language models (LLM) into a detoxification rewritter. We model the fine-tuning process as a Stackelberg game between an LLM (leader) and a toxicity screener (follower), which is a binary style classifier (toxic or non-toxic). The LLM aims to align its preference according to the screener and generate paraphases passing the screening. The primary challenge of non-parallel data fine-tuning is incomplete preference. In the case of unsuccessful paraphrases, the classifier cannot establish a preference between the input and paraphrase, as they belong to the same toxic style. Hence, preference-alignment fine-tuning methods, such as direct preference optimization (DPO), no longer apply. To address the challenge of incomplete preference, we propose Stackelberg response optimization (SRO), adapted from DPO, to enable the LLM to learn from the follower's response. The gist is that SRO decreases the likelihood of generating the paraphrase if it fails the follower's screening while performing DPO on the pair of the toxic input and its paraphrase when the latter passes the screening. Experiments indicate that the SRO-fine-tunned LLM achieves satisfying performance comparable to state-of-the-art models regarding style accuracy, content similarity, and fluency. The overall detoxification performance surpasses other computing methods and matches the human reference. Additional empirical evidence suggests that SRO is sensitive to the screener's feedback, and a slight perturbation leads to a significant performance drop. We release the code and LLM models at \url{<a class="link-external link-https" href="https://github.com/XXXinhong/Detoxification_LLM" rel="external noopener nofollow">this https URL</a>}.

Exploring Methods for Cross-lingual Text Style Transfer: The Case of Text Detoxification

Exploring Cross-lingual Textual Style Transfer with Large Multilingual Language Models

Methods for Detoxification of Texts for the Russian Language

Text Detoxification as Style Transfer in English and Hindi

MultiParaDetox: Extending Text Detoxification with Parallel Data to New Languages

Text Detoxification using Large Pre-trained Neural Models

Russian Texts Detoxification with Levenshtein Editing

Multilingual Text Detoxification Using Google Cloud Translation and Post-Processing

Learning from Response not Preference: A Stackelberg Approach for LLM Detoxification using Non-parallel Data

DiffuDetox: A Mixed Diffusion Model for Text Detoxification

UATST: Towards Unpaired Arbitrary Text-Guided Style Transfer with Cross-Space Modulation

DetoxLLM: A Framework for Detoxification with Explanations

A Multilingual Text Detoxification Method Based on Few-shot Learning and CO-STAR Framework

Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts

SmurfCat at PAN 2024 TextDetox: Alignment of Multilingual Transformers for Text Detoxification

GPT-DETOX: An In-Context Learning-Based Paraphraser for Text Detoxification

Ukrainian Texts Classification: Exploration of Cross-lingual Knowledge Transfer Approaches

Mitigating Text Toxicity with Counterfactual Generation

Transductive Learning for Unsupervised Text Style Transfer

Parameter-Efficient Detoxification with Contrastive Decoding

Challenges in Detoxifying Language Models