Abstract:Text detoxification, a variant of style transfer tasks, finds useful applications in online social media. This work presents a fine-tuning method that only uses non-parallel data to turn large language models (LLM) into a detoxification rewritter. We model the fine-tuning process as a Stackelberg game between an LLM (leader) and a toxicity screener (follower), which is a binary style classifier (toxic or non-toxic). The LLM aims to align its preference according to the screener and generate paraphases passing the screening. The primary challenge of non-parallel data fine-tuning is incomplete preference. In the case of unsuccessful paraphrases, the classifier cannot establish a preference between the input and paraphrase, as they belong to the same toxic style. Hence, preference-alignment fine-tuning methods, such as direct preference optimization (DPO), no longer apply. To address the challenge of incomplete preference, we propose Stackelberg response optimization (SRO), adapted from DPO, to enable the LLM to learn from the follower's response. The gist is that SRO decreases the likelihood of generating the paraphrase if it fails the follower's screening while performing DPO on the pair of the toxic input and its paraphrase when the latter passes the screening. Experiments indicate that the SRO-fine-tunned LLM achieves satisfying performance comparable to state-of-the-art models regarding style accuracy, content similarity, and fluency. The overall detoxification performance surpasses other computing methods and matches the human reference. Additional empirical evidence suggests that SRO is sensitive to the screener's feedback, and a slight perturbation leads to a significant performance drop. We release the code and LLM models at \url{<a class="link-external link-https" href="https://github.com/XXXinhong/Detoxification_LLM" rel="external noopener nofollow">this https URL</a>}.

A Multilingual Text Detoxification Method Based on Few-shot Learning and CO-STAR Framework

Multilingual Text Detoxification Using Google Cloud Translation and Post-Processing

SmurfCat at PAN 2024 TextDetox: Alignment of Multilingual Transformers for Text Detoxification

Exploring Methods for Cross-lingual Text Style Transfer: The Case of Text Detoxification

DESTEIN: Navigating Detoxification of Language Models via Universal Steering Pairs and Head-wise Activation Fusion

A Study of Multilingual Toxic Text Detection Approaches under Imbalanced Sample Distribution

Text Detoxification as Style Transfer in English and Hindi

Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts

Learning from Response not Preference: A Stackelberg Approach for LLM Detoxification using Non-parallel Data

Exploring Cross-lingual Textual Style Transfer with Large Multilingual Language Models

CMD: a framework for Context-aware Model self-Detoxification

GPT-DETOX: An In-Context Learning-Based Paraphraser for Text Detoxification

Self-Detoxifying Language Models via Toxification Reversal

Simple Text Detoxification by Identifying a Linear Toxic Subspace in Language Model Embeddings

Text Detoxification using Large Pre-trained Neural Models

MultiParaDetox: Extending Text Detoxification with Parallel Data to New Languages

Methods for Detoxification of Texts for the Russian Language

Challenges in Detoxifying Language Models

Fine-grained detoxification framework via instance-level prefixes for large language models

Enhancing Multilingual Voice Toxicity Detection with Speech-Text Alignment

Russian Texts Detoxification with Levenshtein Editing