Privacy-Preserving Instructions for Aligning Large Language Models

Da Yu,Peter Kairouz,Sewoong Oh,Zheng Xu

2024-07-02

Abstract:Service providers of large language model (LLM) applications collect user instructions in the wild and use them in further aligning LLMs with users' intentions. These instructions, which potentially contain sensitive information, are annotated by human workers in the process. This poses a new privacy risk not addressed by the typical private optimization. To this end, we propose using synthetic instructions to replace real instructions in data annotation and model fine-tuning. Formal differential privacy is guaranteed by generating those synthetic instructions using privately fine-tuned generators. Crucial in achieving the desired utility is our novel filtering algorithm that matches the distribution of the synthetic instructions to that of the real ones. In both supervised fine-tuning and reinforcement learning from human feedback, our extensive experiments demonstrate the high utility of the final set of synthetic instructions by showing comparable results to real instructions. In supervised fine-tuning, models trained with private synthetic instructions outperform leading open-source models such as Vicuna.

Cryptography and Security,Computation and Language

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper primarily focuses on the privacy risks encountered when handling user instructions in the application of large language models (LLMs) and proposes a novel solution to protect user privacy. Specifically:

1. **Privacy Risks**:
- **Privacy Risk I during Annotation**: User-provided instructions may contain sensitive information, which can be accessed by human annotators during the annotation process.
- **Privacy Risk II during Training**: The trained model may memorize sensitive information from the training data and inadvertently leak this information during deployment.

2. **Limitations of Existing Methods**:
- Commonly used methods, such as detecting and removing personally identifiable information (PII) through manual or automated tools, have the following issues: exposing sensitive information to annotators, high false positive rates of automated tools, and the potential for undetected content to leak user identity when combined with other information.

3. **Proposed New Method**:
- The paper proposes a method based on differential privacy (DP) to generate synthetic instructions as a substitute for real user instructions for annotation and fine-tuning. This method includes two stages:
- Stage 1: Fine-tuning a pre-trained model using differential privacy techniques to generate initial synthetic instructions.
- Stage 2: Using a novel differential privacy resampling algorithm to select a subset of initial synthetic instructions, making their distribution closer to that of real user instructions.

Experimental validation shows that this method outperforms existing public models in tasks involving supervised fine-tuning and reinforcement learning based on human feedback.