Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems

Zhenting Qi,Hanlin Zhang,Eric Xing,Sham Kakade,Himabindu Lakkaraju

2024-10-07

Abstract:Retrieval-Augmented Generation (RAG) improves pre-trained models by incorporating external knowledge at test time to enable customized adaptation. We study the risk of datastore leakage in Retrieval-In-Context RAG Language Models (LMs). We show that an adversary can exploit LMs' instruction-following capabilities to easily extract text data verbatim from the datastore of RAG systems built with instruction-tuned LMs via prompt injection. The vulnerability exists for a wide range of modern LMs that span Llama2, Mistral/Mixtral, Vicuna, SOLAR, WizardLM, Qwen1.5, and Platypus2, and the exploitability exacerbates as the model size scales up. We also study multiple effects of RAG setup on the extractability of data, indicating that following unexpected instructions to regurgitate data can be an outcome of failure in effectively utilizing contexts for modern LMs, and further show that such vulnerability can be greatly mitigated by position bias elimination strategies. Extending our study to production RAG models GPTs, we design an attack that can cause datastore leakage with a 100% success rate on 25 randomly selected customized GPTs with at most 2 queries, and we extract text data verbatim at a rate of 41% from a book of 77,000 words and 3% from a corpus of 1,569,000 words by prompting the GPTs with only 100 queries generated by themselves.

Computation and Language,Artificial Intelligence,Cryptography and Security,Machine Learning

What problem does this paper attempt to address?

The paper attempts to address the issue in Retrieval-Augmented Generation (RAG) systems where Language Models (LMs) may leak information from data storage by following malicious instructions through the Retrieval-In-Context (RIC) method. Specifically, the paper explores how to exploit the instruction-following capability of LMs by injecting malicious prompts to extract text data from the non-parametric data storage of RAG systems. The study finds that this vulnerability is widespread in various modern LMs and that the risk of this vulnerability increases with the scale of the model.

The paper validates the existence of this issue through experiments and analyzes the impact of different factors on the risk of data leakage, including model size, instruction tuning, and whether the content of the data storage was seen during the pre-training phase. Additionally, the paper proposes several mitigation strategies to reduce the risk of data leakage, including Safety-Aware Prompts and Position Bias Elimination. These strategies aim to enhance the model's ability to distinguish between legitimate and malicious prompts, thereby reducing the risk of malicious exploitation.