Abstract:Existing regulations prohibit model developers from accessing protected attributes (gender, race, etc.), often resulting in fairness assessments on populations without knowing their protected groups. In such scenarios, institutions often adopt a separation between the model developers (who train models with no access to the protected attributes) and a compliance team (who may have access to the entire dataset for auditing purposes). However, the model developers might be allowed to test their models for bias by querying the compliance team for group fairness metrics. In this paper, we first demonstrate that simply querying for fairness metrics, such as statistical parity and equalized odds can leak the protected attributes of individuals to the model developers. We demonstrate that there always exist strategies by which the model developers can identify the protected attribute of a targeted individual in the test dataset from just a single query. In particular, we show that one can reconstruct the protected attributes of all the individuals from O(Nk \log( n /Nk)) queries when Nk<<n using techniques from compressed sensing (n: size of the test dataset, Nk: size of smallest group). Our results pose an interesting debate in algorithmic fairness: should querying for fairness metrics be viewed as a neutral-valued solution to ensure compliance with regulations? Or, does it constitute a violation of regulations and privacy if the number of queries answered is enough for the model developers to identify the protected attributes of specific individuals? To address this supposed violation, we also propose Attribute-Conceal, a novel technique that achieves differential privacy by calibrating noise to the smooth sensitivity of our bias query, outperforming naive techniques such as the Laplace mechanism. We also include experimental results on the Adult dataset and synthetic data.

CASTLE: Enhancing the Utility of Inequality Query Auditing Without Denial Threats

Iquery: A Trustworthy and Scalable Blockchain Analytics Platform

Curator Attack: When Blackbox Differential Privacy Auditing Loses Its Power

Can Querying for Bias Leak Protected Attributes? Achieving Privacy With Smooth Sensitivity

Audit Games

A Game-Theoretic Analysis of Auditing Differentially Private Algorithms with Epistemically Disparate Herd

Analysis and Enhancement of a Lattice-Based Data Outsourcing Scheme With Public Integrity Verification

Blockchain-Assisted Conditional Anonymity Privacy-Preserving Public Auditing Scheme With Reward Mechanism

Identity-based controlled delegated outsourcing data integrity auditing scheme

Auditing Private Prediction

An Adaptive Mechanism for Accurate Query Answering under Differential Privacy

Identity-based remote data integrity auditing from lattices for secure cloud storage

Efficient Privacy-Preserving Certificateless Public Auditing of Data in Cloud Storage

Enabling Anonymous Authorized Auditing over Keyword-Based Searchable Ciphertexts in Cloud Storage Systems

Auditing Interval-Based Inference

Efficient Privacy-Preserving Skyline Queries over Outsourced Cloud

Privacy-Preserving Public Auditing for Data Storage Security in Cloud Computing

Certificateless public auditing scheme with designated verifier and privacy-preserving property in cloud storage

ServeDB: Secure, Verifiable, and Efficient Range Queries on Outsourced Database

Certificateless privacy preserving public auditing for dynamic shared data with group user revocation in cloud storage

Dynamic Audit Services for Outsourced Storages in Clouds