Abstract:Differential privacy (DP), as a rigorous mathematical definition quantifying privacy leakage, has become a well-accepted standard for privacy protection. Combined with powerful machine learning (ML) techniques, differentially private machine learning (DPML) is increasingly important. As the most classic DPML algorithm, DP-SGD incurs a significant loss of utility, which hinders DPML's deployment in practice. Many studies have recently proposed improved algorithms based on DP-SGD to mitigate utility loss. However, these studies are isolated and cannot comprehensively measure the performance of improvements proposed in algorithms. More importantly, there is a lack of comprehensive research to compare improvements in these DPML algorithms across utility, defensive capabilities, and generalizability. We fill this gap by performing a holistic measurement of improved DPML algorithms on utility and defense capability against membership inference attacks (MIAs) on image classification tasks. We first present a taxonomy of where improvements are located in the ML life cycle. Based on our taxonomy, we jointly perform an extensive measurement study of the improved DPML algorithms, over twelve algorithms, four model architectures, four datasets, two attacks, and various privacy budget configurations. We also cover state-of-the-art label differential privacy (Label DP) algorithms in the evaluation. According to our empirical results, DP can effectively defend against MIAs, and sensitivity-bounding techniques such as per-sample gradient clipping play an important role in defense. We also explore some improvements that can maintain model utility and defend against MIAs more effectively. Experiments show that Label DP algorithms achieve less utility loss but are fragile to MIAs. ML practitioners may benefit from these evaluations to select appropriate algorithms. To support our evaluation, we implement a modular re-usable software, DPMLBench,(1) which enables sensitive data owners to deploy DPML algorithms and serves as a benchmark tool for researchers and practitioners.

CODER: Protecting Privacy in Image Retrieval with Differential Privacy

Preserving privacy while revealing thumbnail for content-based encrypted image retrieval in the cloud

Comparative Evaluations of Privacy on Digital Images

DP-Image: Differential Privacy for Image Data in Feature Space

DPMLBench: Holistic Evaluation of Differentially Private Machine Learning

CODER: Coupled Diversity-Sensitive Momentum Contrastive Learning for Image-Text Retrieval

Masked image: Visually protected image dataset privacy-preserving scheme for convolutional neural networks

Privacy-friendly Photo Capturing and Sharing System.

Utility-Enhanced Image Obfuscation with Block Differential Privacy

Cloak of Invisibility: Privacy-Friendly Photo Capturing and Sharing System.

PAC Privacy Preserving Diffusion Models

A Privacy-Preserving Image Retrieval Scheme Based Secure Knn, DNA Coding and Deep Hashing

Privacy in Cloud Computing through Immersion-based Coding

Controllable Identity De-Identification for Face Image

Enhancing User-Centric Privacy Protection: An Interactive Framework through Diffusion Models and Machine Unlearning

PPVF: An Efficient Privacy-Preserving Online Video Fetching Framework with Correlated Differential Privacy

Privacy-preserving outsourcing of image global feature detection

TDHPPIR: An Efficient Deep Hashing Based Privacy-Preserving Image Retrieval Method

PrivImage: Differentially Private Synthetic Image Generation using Diffusion Models with Semantic-Aware Pretraining

Unlocking Accuracy and Fairness in Differentially Private Image Classification

Human-Unrecognizable Differential Private Noised Image Generation Method