Abstract:We study private empirical risk minimization (ERM) problem for losses satisfying the $(\gamma,\kappa)$-Kurdyka-Łojasiewicz (KL) condition. The Polyak-Łojasiewicz (PL) condition is a special case of this condition when $\kappa=2$. Specifically, we study this problem under the constraint of $\rho$ zero-concentrated differential privacy (zCDP). When $\kappa\in[1,2]$ and the loss function is Lipschitz and smooth over a sufficiently large region, we provide a new algorithm based on variance reduced gradient descent that achieves the rate $\tilde{O}\big(\big(\frac{\sqrt{d}}{n\sqrt{\rho}}\big)^\kappa\big)$ on the excess empirical risk, where $n$ is the dataset size and $d$ is the dimension. We further show that this rate is nearly optimal. When $\kappa \geq 2$ and the loss is instead Lipschitz and weakly convex, we show it is possible to achieve the rate $\tilde{O}\big(\big(\frac{\sqrt{d}}{n\sqrt{\rho}}\big)^\kappa\big)$ with a private implementation of the proximal point method. When the KL parameters are unknown, we provide a novel modification and analysis of the noisy gradient descent algorithm and show that this algorithm achieves a rate of $\tilde{O}\big(\big(\frac{\sqrt{d}}{n\sqrt{\rho}}\big)^{\frac{2\kappa}{4-\kappa}}\big)$ adaptively, which is nearly optimal when $\kappa = 2$. We further show that, without assuming the KL condition, the same gradient descent algorithm can achieve fast convergence to a stationary point when the gradient stays sufficiently large during the run of the algorithm. Specifically, we show that this algorithm can approximate stationary points of Lipschitz, smooth (and possibly nonconvex) objectives with rate as fast as $\tilde{O}\big(\frac{\sqrt{d}}{n\sqrt{\rho}}\big)$ and never worse than $\tilde{O}\big(\big(\frac{\sqrt{d}}{n\sqrt{\rho}}\big)^{1/2}\big)$. The latter rate matches the best known rate for methods that do not rely on variance reduction.

Differentially private empirical risk minimization for AUC maximization

Differentially Private Empirical Risk Minimization

Differentially Private Empirical Risk Minimization Revisited: Faster and More General

ppAURORA: Privacy Preserving Area Under Receiver Operating Characteristic and Precision-Recall Curves

Accuracy First: Selecting a Differential Privacy Level for Accuracy-Constrained ERM

A Differentially Private Weighted Empirical Risk Minimization Procedure and its Application to Outcome Weighted Learning

Differentially Private Algorithms for Empirical Machine Learning

Distributed empirical risk minimization with differential privacy

Differentially Private Non-Convex Optimization under the KL Condition with Optimal Rates

Efficient Private ERM for Smooth Objectives.

Stochastic Hard Thresholding Algorithms for AUC Maximization

Differentially Private Confidence Intervals for Empirical Risk Minimization.

The Power of Sampling: Dimension-free Risk Bounds in Private ERM

The Cost of Privacy: Optimal Rates of Convergence for Parameter Estimation with Differential Privacy

Differentially Private Optimization for Smooth Nonconvex ERM

Towards Understanding the Fairness of Differentially Private Margin Classifiers

Differentially private regression analysis with dynamic privacy allocation

Error Density-dependent Empirical Risk Minimization

An efficient variance estimator of AUC and its applications to binary classification

Differentially private Riemannian optimization

Universal Rates of Empirical Risk Minimization