Abstract:Differentially Private Stochastic Gradient Descent (DP-SGD) is a popular iterative algorithm used to train machine learning models while formally guaranteeing the privacy of users. However, the privacy analysis of DP-SGD makes the unrealistic assumption that all intermediate iterates (aka internal state) of the algorithm are released since, in practice, only the final trained model, i.e., the final iterate of the algorithm is released. In this hidden state setting, prior work has provided tighter analyses, albeit only when the loss function is constrained, e.g., strongly convex and smooth or linear. On the other hand, the privacy leakage observed empirically from hidden state DP-SGD, even when using non-convex loss functions, suggests that there is in fact a gap between the theoretical privacy analysis and the privacy guarantees achieved in practice. Therefore, it remains an open question whether hidden state privacy amplification for DP-SGD is possible for all (possibly non-convex) loss functions in general. In this work, we design a counter-example and show, both theoretically and empirically, that a hidden state privacy amplification result for DP-SGD for all loss functions in general is not possible. By carefully constructing a loss function for DP-SGD, we show that for specific loss functions, the final iterate of DP-SGD alone leaks as much information as the sequence of all iterates combined. Furthermore, we empirically verify this result by evaluating the privacy leakage from the final iterate of DP-SGD with our loss function and show that this exactly matches the theoretical upper bound guaranteed by DP. Therefore, we show that the current privacy analysis for DP-SGD is tight for general loss functions and conclude that no privacy amplification is possible for DP-SGD in general for all (possibly non-convex) loss functions.

Privacy Amplification for Matrix Mechanisms

Near Exact Privacy Amplification for Matrix Mechanisms

Multi-Epoch Matrix Factorization Mechanisms for Private Machine Learning

Improved Matrix Gaussian Mechanism for Differential Privacy

A Differential Privacy Mechanism Design Under Matrix-Valued Query

Privacy Amplification for the Gaussian Mechanism via Bounded Support

Improved Differential Privacy for SGD via Optimal Private Linear Operators on Adaptive Streams

Reducing Noise Level in Differential Privacy through Matrix Masking

Unified Mechanism-Specific Amplification by Subsampling and Group Privacy Amplification

Matrix Gaussian Mechanisms for Differentially-Private Learning

Beyond the Calibration Point: Mechanism Comparison in Differential Privacy

On the Privacy of Selection Mechanisms with Gaussian Noise

Gradient Descent with Linearly Correlated Noise: Theory and Applications to Differential Privacy

Private Alternating Least Squares: Practical Private Matrix Completion with Tighter Rates

How Private are DP-SGD Implementations?

Privacy Profiles for Private Selection

Learning Numeric Optimal Differentially Private Truncated Additive Mechanisms

It's Our Loss: No Privacy Amplification for Hidden State DP-SGD With Non-Convex Loss

Less is More: Revisiting the Gaussian Mechanism for Differential Privacy

Differential Privacy with Higher Utility by Exploiting Coordinate-wise Disparity: Laplace Mechanism Can Beat Gaussian in High Dimensions

DMM: Distributed Matrix Mechanism for Differentially-Private Federated Learning using Packed Secret Sharing