Abstract:Label differential privacy (label-DP) is a popular framework for training private ML models on datasets with public features and sensitive private labels. Despite its rigorous privacy guarantee, it has been observed that in practice label-DP does not preclude label inference attacks (LIAs): Models trained with label-DP can be evaluated on the public training features to recover, with high accuracy, the very private labels that it was designed to protect. In this work, we argue that this phenomenon is not paradoxical and that label-DP is designed to limit the advantage of an LIA adversary compared to predicting training labels using the Bayes classifier. At label-DP $\epsilon=0$ this advantage is zero, hence the optimal attack is to predict according to the Bayes classifier and is independent of the training labels. Our bound shows the semantic protection conferred by label-DP and gives guidelines on how to choose $\varepsilon$ to limit the threat of LIAs below a certain level. Finally, we empirically demonstrate that our result closely captures the behavior of simulated attacks on both synthetic and real world datasets.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to explore whether label - differential privacy (label - DP) can prevent label - inference attacks (LIAs). Specifically, the paper analyzes the following issues: 1. **Limitations of label - differential privacy**: Although label - differential privacy provides a certain level of privacy protection, in practice, models trained with label - differential privacy may still be used to infer sensitive labels that should have been protected. This phenomenon has raised doubts about the effectiveness of label - differential privacy. 2. **Feasibility of label - inference attacks**: The paper studies the success rate of label - inference attacks under different privacy parameters $\epsilon$, and explores why models may still leak a large amount of label information even under strict label - differential privacy guarantees. 3. **The relationship between label - differential privacy and Bayesian classifiers**: The paper points out that the design goal of label - differential privacy is not to completely prevent label - inference attacks, but to limit the advantage of attackers relative to predicting training labels using Bayesian classifiers. When $\epsilon = 0$, this advantage is zero, that is, the optimal attack is to predict according to the Bayesian classifier, regardless of the training labels. 4. **How to choose an appropriate $\epsilon$ value**: The paper proposes a theoretical framework for evaluating and selecting an appropriate $\epsilon$ value to keep the risk of label - inference attacks below a certain level. 5. **Experimental verification**: Through experiments on simulated datasets and real - world datasets, the validity of the theoretical results is verified, and the performance of label - differential privacy in different scenarios is demonstrated. ### Main conclusions of the paper - Label - differential privacy cannot completely prevent label - inference attacks at arbitrarily small $\epsilon$ values. - The design goal of label - differential privacy is to limit the advantage of attackers relative to Bayesian classifiers, rather than absolutely limit the success rate of label - inference attacks. - By choosing an appropriate $\epsilon$ value, the success rate of label - inference attacks can be effectively reduced. ### Formula summary - **Expected Attack Utility (EAU)**: \[ \text{EAU}(A, K)=\mathbb{E}_{y, M}\left[\frac{1}{n}\sum_{i = 1}^{n}u(\hat{y}_{i}, y_{i})\mid X\right] \] - **Label - Independent Expected Attack Utility (L - EAU)**: \[ \text{L - EAU}(K)=\frac{1}{n}\sum_{i = 1}^{n}\max_{y\in Y}\mathbb{E}_{y_{i}}[u(y, y_{i})\mid X_{i}] \] - **Advantage**: \[ \text{Adv}(A, K)=\text{EAU}(A, K)-\text{L - EAU}(K) \] - **Upper bound of the attack advantage of label - differential privacy**: \[ \text{Adv}(A, K)\leq\left(1-\frac{2}{1 + e^{\epsilon(1-\delta)}}\right)\cdot B \] where $B$ is the upper bound of the attack utility function $u$. Through these formulas, the paper provides theoretical guarantees for label - differential privacy under different privacy parameters and explains why models may still face the risk of label - inference attacks even under strict privacy guarantees.

Does Label Differential Privacy Prevent Label Inference Attacks?

DPMLBench: Holistic Evaluation of Differentially Private Machine Learning

Label differential privacy via clustering

Does Differential Privacy Prevent Backdoor Attacks in Practice?

Label Differential Privacy via Aggregation

Enhancing Learning with Label Differential Privacy by Vector Approximation

Auditing Privacy Mechanisms via Label Inference Attacks

Similarity-based Label Inference Attack against Training and Inference of Split Learning

Privacy-Preserving Federated Learning Against Label-Flipping Attacks on Non-IID Data

The Limits of Differential Privacy in Online Learning

Privacy-Preserving Student Learning with Differentially Private Data-Free Distillation

Auditing Private Prediction

Attackers Are Not the Same! Unveiling the Impact of Feature Distribution on Label Inference Attacks

One Parameter Defense -- Defending against Data Inference Attacks via Differential Privacy

Does Differential Privacy Impact Bias in Pretrained NLP Models?

DPCL: Contrastive Representation Learning with Differential Privacy

Differential Privacy Protection Against Membership Inference Attack on Machine Learning for Genomic Data

Label-Only Membership Inference Attack Based on Model Explanation

Be Careful What You Smooth For: Label Smoothing Can Be a Privacy Shield but Also a Catalyst for Model Inversion Attacks

Why Does Differential Privacy with Large Epsilon Defend Against Practical Membership Inference Attacks?