Abstract:Because machine learning models, especially black-box malicious models vulnerable to attribute inference attacks, are capable of generating a great deal of privacy leakage, recent work has focused on assessing these models in an attempt to prevent unexpected attribute privacy leakage. While there has been some success at model privacy risk evaluations, these traditional solutions are almost brittle in practice because they not only require white-box access to obtain model feature layer outputs but also their evaluation results are heavily influenced by the training dataset and the model structure, leading to difficulty in generalization. In this paper, we propose a novel unawareness detection mechanism for discovering black-box malicious models and quantifying potential unawareness privacy leakage risk along with machine learning models to overcome the two limitations. A new method for quantifying the privacy risk caused by a specific loss function has been proposed to mitigate the impact of the training dataset and the model structure. A new evaluation model has also been proposed that uses Matthew's correlation coefficient score as a new metric and the final output of the target model as a new input. In addition, the theoretical upper bound of the model privacy risk has also been given a mathematical formula that is positively correlated to the mutual information between the sensitive attributes and the target model outputs. Compared with traditional detection methods, our evaluation model reduces the requirement for model access and minimizes evaluation errors caused by data imbalance, and our privacy risk assessment method and theoretical upper bounds on privacy risk can be applied to a broader range of datasets and target model structures. The experimental results show that the adversary's prediction capability is affected by the distribution of datasets and the level of malicious intent of the model, which is consistent with the theoretical prediction, and the detecting method can find potential model privacy leaks in public datasets UTKFace and FairFace.

Unraveling Privacy Threat Modeling Complexity: Conceptual Privacy Analysis Layers

How to Model Privacy Threats in the Automotive Domain

A Model-oriented Reasoning Framework for Privacy Analysis of Complex Systems

Advanced Cloud Privacy Threat Modeling

The Privacy Pillar -- A Conceptual Framework for Foundation Model-based Systems

Design and Implementation of the Advanced Cloud Privacy Threat Modeling

Privacy in Foundation Models: A Conceptual Framework for System Design

A Threat Model for Soft Privacy on Smart Cars

Privacy Patterns

Up-to-date Threat Modelling for Soft Privacy on Smart Cars

Threat models over space and time: A case study of end‐to‐end‐encrypted messaging applications

Privacy Risks of Social Interaction Structure: Network Learning in Quadratic Games

Buying Privacy: User Perceptions of Privacy Threats from Mobile Apps

Modeling Privacy Leakage Risks in Large-Scale Social Networks

Unawareness detection: Discovering black-box malicious models and quantifying privacy leakage risks

Modeling of Personalized Privacy Disclosure Behavior: A Formal Method Approach

An Overview of Privacy in Machine Learning

Analysis of Human Awareness of Security and Privacy Threats in Smart Environments

Security and Privacy Product Inclusion

Information privacy and correlates: an empirical attempt to bridge and distinguish privacy-related concepts

Privacy Side Channels in Machine Learning Systems