Abstract:Designing privacy-preserving machine learning algorithms has received great attention in recent years, especially in the setting when the data contains sensitive information. Differential privacy (DP) is a widely used mechanism for data analysis with privacy guarantees. In this paper, we produce a differentially private random feature model. Random features, which were proposed to approximate large-scale kernel machines, have been used to study privacy-preserving kernel machines as well. We consider the over-parametrized regime (more features than samples) where the non-private random feature model is learned via solving the min-norm interpolation problem, and then we apply output perturbation techniques to produce a private model. We show that our method preserves privacy and derive a generalization error bound for the method. To the best of our knowledge, we are the first to consider privacy-preserving random feature models in the over-parametrized regime and provide theoretical guarantees. We empirically compare our method with other privacy-preserving learning methods in the literature as well. Our results show that our approach is superior to the other methods in terms of generalization performance on synthetic data and benchmark data sets. Additionally, it was recently observed that DP mechanisms may exhibit and exacerbate disparate impact, which means that the outcomes of DP learning algorithms vary significantly among different groups. We show that both theoretically and empirically, random features have the potential to reduce disparate impact, and hence achieve better fairness.

Differentially Private SGD with Random Features

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Differentially private stochastic gradient descent with low-noise

Improving Differentially Private SGD via Randomly Sparsified Gradients

A(DP)$^2$2SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Individual Privacy Accounting for Differentially Private Stochastic Gradient Descent

DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learning

Differentially Private Random Feature Model

Bolt-on Differential Privacy for Scalable Stochastic Gradient Descent-based Analytics

Differentially Private Learning with Small Public Data.

An Improved Privacy-Preserving Stochastic Gradient Descent Algorithm

On the Privacy of Noisy Stochastic Gradient Descent for Convex Optimization

Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification

The importance of feature preprocessing for differentially private linear optimization

Large Scale Private Learning Via Low-rank Reparametrization

Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach

Gradients Look Alike: Sensitivity is Often Overestimated in DP-SGD

Auditing Differentially Private Machine Learning: How Private is Private SGD?

DP-LSSGD: A Stochastic Optimization Method to Lift the Utility in Privacy-Preserving ERM

Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weight

Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds