Abstract:We study the problem of $(\epsilon,\delta)$-differentially private learning of linear predictors with convex losses. We provide results for two subclasses of loss functions. The first case is when the loss is smooth and non-negative but not necessarily Lipschitz (such as the squared loss). For this case, we establish an upper bound on the excess population risk of $\tilde{O}\left(\frac{\Vert w^*\Vert}{\sqrt{n}} + \min\left\{\frac{\Vert w^* \Vert^2}{(n\epsilon)^{2/3}},\frac{\sqrt{d}\Vert w^*\Vert^2}{n\epsilon}\right\}\right)$, where $n$ is the number of samples, $d$ is the dimension of the problem, and $w^*$ is the minimizer of the population risk. Apart from the dependence on $\Vert w^\ast\Vert$, our bound is essentially tight in all parameters. In particular, we show a lower bound of $\tilde{\Omega}\left(\frac{1}{\sqrt{n}} + {\min\left\{\frac{\Vert w^*\Vert^{4/3}}{(n\epsilon)^{2/3}}, \frac{\sqrt{d}\Vert w^*\Vert}{n\epsilon}\right\}}\right)$. We also revisit the previously studied case of Lipschitz losses [SSTT20]. For this case, we close the gap in the existing work and show that the optimal rate is (up to log factors) $\Theta\left(\frac{\Vert w^*\Vert}{\sqrt{n}} + \min\left\{\frac{\Vert w^*\Vert}{\sqrt{n\epsilon}},\frac{\sqrt{\text{rank}}\Vert w^*\Vert}{n\epsilon}\right\}\right)$, where $\text{rank}$ is the rank of the design matrix. This improves over existing work in the high privacy regime. Finally, our algorithms involve a private model selection approach that we develop to enable attaining the stated rates without a-priori knowledge of $\Vert w^*\Vert$.

Differentially Private Model Personalization

Differentially Private Variational Inference for Non-conjugate Models

Differentially Private Model Selection with Penalized and Constrained Likelihood

Training Private Models That Know What They Don't Know

Differentially-Private Deep Learning from an Optimization Perspective.

Optimal Differentially Private Model Training with Public Data

Not one but many Tradeoffs: Privacy Vs. Utility in Differentially Private Machine Learning

Individual Privacy Accounting for Differentially Private Stochastic Gradient Descent

Evaluating Differentially Private Machine Learning in Practice

Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning

Improving Differentially Private Models with Active Learning

Differentially Private Learning Beyond the Classical Dimensionality Regime

Differentially Private Learning with Small Public Data.

Fine-Tuning Large Language Models with User-Level Differential Privacy

When Does Differentially Private Learning Not Suffer in High Dimensions?

Differentially Private Generalized Linear Models Revisited

Differentially Private Algorithms for Empirical Machine Learning

Differential Privacy in Personalized Pricing with Nonparametric Demand Models

Better Private Linear Regression Through Better Private Feature Selection

Differentially Private Simple Linear Regression

Towards Sharper Utility Bounds for Differentially Private Pairwise Learning