Abstract:The rapid development of large language models (LLMs), like ChatGPT, has resulted in the widespread presence of LLM-generated content on social media platforms, raising concerns about misinformation, data biases, and privacy violations, which can undermine trust in online discourse. While detecting LLM-generated content is crucial for mitigating these risks, current methods often focus on binary classification, failing to address the complexities of real-world scenarios like human-AI collaboration. To move beyond binary classification and address these challenges, we propose a new paradigm for detecting LLM-generated content. This approach introduces two novel tasks: LLM Role Recognition (LLM-RR), a multi-class classification task that identifies specific roles of LLM in content generation, and LLM Influence Measurement (LLM-IM), a regression task that quantifies the extent of LLM involvement in content creation. To support these tasks, we propose LLMDetect, a benchmark designed to evaluate detectors' performance on these new tasks. LLMDetect includes the Hybrid News Detection Corpus (HNDC) for training detectors, as well as DetectEval, a comprehensive evaluation suite that considers five distinct cross-context variations and multi-intensity variations within the same LLM role. This allows for a thorough assessment of detectors' generalization and robustness across diverse contexts. Our empirical validation of 10 baseline detection methods demonstrates that fine-tuned PLM-based models consistently outperform others on both tasks, while advanced LLMs face challenges in accurately detecting their own generated content. Our experimental results and analysis offer insights for developing more effective detection models for LLM-generated content. This research enhances the understanding of LLM-generated content and establishes a foundation for more nuanced detection methodologies.

Learning to Rewrite: Generalized LLM-Generated Text Detection

Unveiling Large Language Models Generated Texts: A Multi-Level Fine-Grained Detection Framework

Beat LLMs at Their Own Game: Zero-Shot LLM-Generated Text Detection Via Querying ChatGPT.

The Science of Detecting LLM-Generated Texts

Beyond Binary: Towards Fine-Grained LLM-Generated Text Detection via Role Recognition and Involvement Measurement

Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting

RewriteLM: An Instruction-Tuned Large Language Model for Text Rewriting

Robust Detection of LLM-Generated Text: A Comparative Analysis

Improving Logits-based Detector without Logits from Black-box LLMs

LLM-Detector: Improving AI-Generated Chinese Text Detection with Open-Source LLM Instruction Tuning

Deciphering Textual Authenticity: A Generalized Strategy through the Lens of Large Language Semantics for Detecting Human vs. Machine-Generated Text

SMLT-MUGC: Small, Medium, and Large Texts -- Machine versus User-Generated Content Detection and Comparison

DALD: Improving Logits-based Detector without Logits from Black-box LLMs

DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios

Enhancing Robustness of LLM-Synthetic Text Detectors for Academic Writing: A Comprehensive Analysis

Humanizing the Machine: Proxy Attacks to Mislead LLM Detectors

Towards Reliable Detection of LLM-Generated Texts: A Comprehensive Evaluation Framework with CUDRT

LLM-DetectAIve: a Tool for Fine-Grained Machine-Generated Text Detection

Imitate Before Detect: Aligning Machine Stylistic Preference for Machine-Revised Text Detection

CUDRT: Benchmarking the Detection of Human Vs. Large Language Models Generated Texts.

Which LLMs are Difficult to Detect? A Detailed Analysis of Potential Factors Contributing to Difficulties in LLM Text Detection