Abstract:A major public concern regarding the training of large language models (LLMs) is whether they abusing copyrighted online text. Previous membership inference methods may be misled by similar examples in vast amounts of training data. Additionally, these methods are often too complex for general users to understand and use, making them centralized, lacking transparency, and trustworthiness. To address these issues, we propose an alternative \textit{insert-and-detection} methodology, advocating that web users and content platforms employ \textbf{\textit{unique identifiers}} for reliable and independent membership inference. Users and platforms can create their own identifiers, embed them in copyrighted text, and independently detect them in future LLMs. As an initial demonstration, we introduce \textit{ghost sentences}, a primitive form of unique identifiers, consisting primarily of passphrases made up of random words. By embedding one ghost sentences in a few copyrighted texts, users can detect its membership using a perplexity test and a \textit{user-friendly} last-$k$ words test. The perplexity test is based on the fact that LLMs trained on natural language should exhibit high perplexity when encountering unnatural passphrases. As the repetition increases, users can leverage the verbatim memorization ability of LLMs to perform a last-$k$ words test by chatting with LLMs without writing any code. Both tests offer rigorous statistical guarantees for membership inference. For LLaMA-13B, a perplexity test on 30 ghost sentences with an average of 7 repetitions in 148K examples yields a 0.891 ROC AUC. For the last-$k$ words test with OpenLLaMA-3B, 11 out of 16 users, with an average of 24 examples each, successfully identify their data from 1.8M examples.

Measuring Copyright Risks of Large Language Model via Partial Information Probing

Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data?

Evaluating Copyright Takedown Methods for Language Models

Digger: Detecting Copyright Content Mis-usage in Large Language Model Training

Copyright Violations and Large Language Models

SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generation

DE-COP: Detecting Copyrighted Content in Language Models Training Data

Copyright Traps for Large Language Models

CopyLens: Dynamically Flagging Copyrighted Sub-Dataset Contributions to LLM Outputs

Watermarking Large Language Models and the Generated Content: Opportunities and Challenges

Source Attribution for Large Language Model-Generated Data

LLMs and Memorization: On Quality and Specificity of Copyright Compliance

Instructional Fingerprinting of Large Language Models

The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective

LLMs Plagiarize: Ensuring Responsible Sourcing of Large Language Model Training Data Through Knowledge Graph Comparison

How to Protect Copyright Data in Optimization of Large Language Models?

Avoiding Copyright Infringement via Large Language Model Unlearning

CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generation

Turning Your Strength into Watermark: Watermarking Large Language Model via Knowledge Injection

Protecting Copyrighted Material with Unique Identifiers in Large Language Model Training

Matching Pairs: Attributing Fine-Tuned Models to their Pre-Trained Large Language Models