Curator Attack: When Blackbox Differential Privacy Auditing Loses Its Power

Shiming Wang,Liyao Xiang,Bowei Cheng,Zhe Ji,Tianran Sun,Xinbing Wang

DOI: https://doi.org/10.1145/3658644.3690367

2024-11-26

Abstract:A surge in data-driven applications enhances everyday life but also raises serious concerns about private information leakage. Hence many privacy auditing tools are emerging for checking if the data sanitization performed meets the privacy standard of the data owner. Blackbox auditing for differential privacy is particularly gaining popularity for its effectiveness and applicability to a wide range of scenarios. Yet, we identified that blackbox auditing is essentially flawed with its setting: small probabilities or densities are ignored due to inaccurate observation. Our argument is based on a solid false positive analysis from a hypothesis testing perspective, which is missed out by prior blackbox auditing tools. This oversight greatly reduces the reliability of these tools, as it allows malicious or incapable data curators to pass the auditing with an overstated privacy guarantee, posing significant risks to data owners. We demonstrate the practical existence of such threats in classical differential privacy mechanisms against four representative blackbox auditors with experimental validations. Our findings aim to reveal the limitations of blackbox auditing tools, empower the data owner with the awareness of risks in using these tools, and encourage the development of more reliable differential privacy auditing methods.

Cryptography and Security

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the limitations of black - box auditing in differential privacy protection, especially the False Positive (FP) problem caused when auditing tools ignore low - probability events. Specifically, the paper points out that existing black - box auditing tools cannot effectively detect false privacy claims of malicious data managers due to their design flaws - that is, ignoring low - probability or density values in the case of inaccurate observations. This behavior of ignoring low - probability events greatly reduces the reliability of auditing tools, enabling malicious or incompetent data managers to pass the audit, thus posing a serious privacy threat to data owners.

The main contributions of the paper are as follows:
1. **Reveal the inherent weaknesses of black - box auditing tools**: Through a strict analysis of hypothesis testing, the paper points out the deficiencies of existing black - box auditing tools in dealing with low - probability events, which lead to a high false positive rate.
2. **Raise data owners' risk awareness**: The paper aims to make data owners aware of the risks of using existing auditing services and remind them to be more cautious when choosing auditing tools.
3. **Promote the development of more reliable auditing methods**: By demonstrating the limitations of existing tools, the paper encourages the research community to develop more reliable differential privacy auditing methods.

The paper verifies through experiments that in the classical differential privacy mechanism, four representative black - box auditing tools do have the above - mentioned problems, and proposes specific attack scenarios, showing how to use the vulnerabilities of these tools for malicious operations. These findings are of great significance for evaluating the effectiveness of existing black - box auditing tools and also provide directions for future research.