Spotting AI's Touch: Identifying LLM-Paraphrased Spans in Text

Yafu Li,Zhilin Wang,Leyang Cui,Wei Bi,Shuming Shi,Yue Zhang

2024-05-29

Abstract:AI-generated text detection has attracted increasing attention as powerful language models approach human-level generation. Limited work is devoted to detecting (partially) AI-paraphrased texts. However, AI paraphrasing is commonly employed in various application scenarios for text refinement and diversity. To this end, we propose a novel detection framework, paraphrased text span detection (PTD), aiming to identify paraphrased text spans within a text. Different from text-level detection, PTD takes in the full text and assigns each of the sentences with a score indicating the paraphrasing degree. We construct a dedicated dataset, PASTED, for paraphrased text span detection. Both in-distribution and out-of-distribution results demonstrate the effectiveness of PTD models in identifying AI-paraphrased text spans. Statistical and model analysis explains the crucial role of the surrounding context of the paraphrased text spans. Extensive experiments show that PTD models can generalize to versatile paraphrasing prompts and multiple paraphrased text spans. We release our resources at <a class="link-external link-https" href="https://github.com/Linzwcs/PASTED" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

This paper aims to address the fine-grained detection problem of paraphrased text spans partially rewritten by AI. With the development of large language models (LLMs), AI-generated text detection has become a research hotspot. However, existing work mainly focuses on the binary classification problem of whether an entire text is AI-generated. In practical applications, AI rewriting techniques are widely used to polish text or increase diversity, making it particularly important to identify which sentences or spans have been AI-rewritten in a fine-grained manner.

The paper proposes a new task—**Paraphrased Text Span Detection (PTD)**. This method not only identifies which sentences or spans have been AI-rewritten but also quantifies the degree of difference between these rewritten spans and the original text. To support this task, the authors constructed a dataset named PASTED, which contains original texts and their corresponding rewritten versions, and validated the effectiveness of the PTD model through experiments. Additionally, the paper explores the impact of different context-sensitive rewriting methods on detection performance and demonstrates the model's generalization ability in handling different rewriting prompts.