Abstract:Large language models (LLMs) have achieved remarkable performance on a wide range of tasks. However, recent studies have shown that LLMs can memorize training data and simple repeated tokens can trick the model to leak the data. In this paper, we take a step further and show that certain special characters or their combinations with English letters are stronger memory triggers, leading to more severe data leakage. The intuition is that, since LLMs are trained with massive data that contains a substantial amount of special characters (e.g. structural symbols {, } of JSON files, and @, # in emails and online posts), the model may memorize the co-occurrence between these special characters and the raw texts. This motivates us to propose a simple but effective Special Characters Attack (SCA) to induce training data leakage. Our experiments verify the high effectiveness of SCA against state-of-the-art LLMs: they can leak diverse training data, such as code corpus, web pages, and personally identifiable information, and sometimes generate non-stop outputs as a byproduct. We further show that the composition of the training data corpus can be revealed by inspecting the leaked data -- one crucial piece of information for pre-training high-performance LLMs. Our work can help understand the sensitivity of LLMs to special characters and identify potential areas for improvement.

What problem does this paper attempt to address?

The problems that this paper attempts to solve are as follows: Large Language Models (LLMs) may remember the training data during the training process, and some special characters or their combinations with English characters can trigger more serious data leakage. Specifically, the paper focuses on how to extract the training data from LLMs through a specific attack method (Special Characters Attack, SCA) and reveal the sensitivity of these models to special characters. ### Main problems and motivations of the paper 1. **Data leakage risks**: - LLMs may remember the specific content when processing a large amount of data. - Simple repeated phrases or specific character sequences can induce the model to leak training data. 2. **Limitations of existing attack methods**: - Previous attack methods mainly relied on repeating words or phrases, but these methods are easy to be detected and defended. - A new attack method is needed to more effectively extract training data and reveal the model's sensitivity to special characters. 3. **Research motivations**: - Understand the memory mechanism of LLMs for special characters. - Propose a new attack method (SCA) that uses special characters and their combinations to induce data leakage. - Analyze the leaked data to reveal the composition of the training data set, thereby evaluating potential security threats. ### Specific problem descriptions The paper proposes a new attack method named Special Characters Attack (SCA), which aims to induce LLMs to leak training data by inputting specific special character sequences. Specific problems include: - **Selection of special characters**: Which special characters are most likely to trigger data leakage? - **Generation of attack sequences**: How to design effective attack sequences to maximize the possibility of data leakage? - **Analysis of model responses**: Analyze the model's responses after receiving special character sequences and identify the types of leaked data. - **Joint memory mechanism**: Explore the joint memory mechanism between special characters and other texts, and explain why special characters can cause data leakage. ### Main contributions 1. **Propose a new attack method**: Special Characters Attack (SCA), which uses special characters and their combinations to induce LLMs to leak training data. 2. **Verify effectiveness**: Verify the effectiveness of SCA on multiple LLMs through experiments, showing that it can leak multiple types of training data. 3. **Reveal training data set information**: By analyzing the leaked data, reveal the composition information of the target LLM's training data set. 4. **Discover potential mechanisms**: Discover the joint memory mechanism between control characters and non - control characters, and explain the reasons for data leakage. Through these studies, the paper not only reveals the sensitivity of LLMs to special characters, but also provides new ideas and methods for future research, helping to improve the security and privacy protection measures of LLMs.

Special Characters Attack: Toward Scalable Training Data Extraction From Large Language Models

Uncovering Safety Risks of Large Language Models through Concept Activation Vector

Decoding Secret Memorization in Code LLMs Through Token-Level Characterization

A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models

Training Data Leakage Analysis in Language Models

A Little Leak Will Sink a Great Ship: Survey of Transparency for Large Language Models from Start to Finish

Extracting Training Data from Large Language Models

Harnessing Task Overload for Scalable Jailbreak Attacks on Large Language Models

Composite Backdoor Attacks Against Large Language Models

Impact of Non-Standard Unicode Characters on Security and Comprehension in Large Language Models

Using Large Language Models for Cybersecurity Capture-The-Flag Challenges and Certification Questions

Unique Security and Privacy Threats of Large Language Model: A Comprehensive Survey

CleanGen: Mitigating Backdoor Attacks for Generation Tasks in Large Language Models

Can LLMs be Fooled? Investigating Vulnerabilities in LLMs

A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly

On Extracting Specialized Code Abilities from Large Language Models: A Feasibility Study

Probing the Safety Response Boundary of Large Language Models via Unsafe Decoding Path Generation

Extracting Memorized Training Data via Decomposition

Data Stealing Attacks against Large Language Models via Backdooring