Differentially Private Optimization for Smooth Nonconvex ERM

Changyu Gao,Stephen J. Wright

2023-06-09

Abstract:We develop simple differentially private optimization algorithms that move along directions of (expected) descent to find an approximate second-order solution for nonconvex ERM. We use line search, mini-batching, and a two-phase strategy to improve the speed and practicality of the algorithm. Numerical experiments demonstrate the effectiveness of these approaches.

Machine Learning,Cryptography and Security,Optimization and Control

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to solve the problem of how to design efficient **differentially private optimization algorithms** to find approximate second - order solutions in the **non - convex empirical risk minimization (ERM) problem**. Specifically, the authors propose several improvement techniques to enhance the practicality and efficiency of these algorithms, including:

1. **Backtracking Line Search**: By adaptively adjusting the step size, the algorithm can better adapt to the local geometric characteristics of the objective function.
2. **Mini - batching**: By using data subsets to accelerate the calculation and reduce the noise impact in each iteration.
3. **Two - phase Strategy**: By using the privacy budget in stages, the convergence speed and accuracy of the algorithm are improved.

#### Background and Motivation

- **Differential Privacy (DP)**: It is a strict privacy protection framework that ensures that sensitive information of individual data will not be leaked during the training process of machine learning models.
- **Non - convex optimization problems**: Many practical problems (such as loss functions in deep learning) are non - convex, and it is very difficult to find the global optimal solution. Therefore, approximate solutions that meet certain conditions are usually sought.
- **Empirical Risk Minimization (ERM)**: It is a classic problem in machine learning. The goal is to find the optimal parameters by minimizing the average loss on the training data.

#### Specific Problem Description

For a given smooth but non - convex function \( f: \mathbb{R}^d \to \mathbb{R} \), consider the following minimization problem:
\[
\min_{w \in \mathbb{R}^d} f(w)
\]
We hope to find an approximate second - order solution \( w \), defined as:
\[
\|\nabla f(w)\| \leq \epsilon_g, \quad \lambda_{\min}(\nabla^2 f(w)) \geq -\epsilon_H
\]
where \( \epsilon_g \) and \( \epsilon_H \) are given positive numbers. In particular, when \( \epsilon_g=\alpha \) and \( \epsilon_H = \sqrt{M\alpha} \), we are seeking a \((\alpha, \sqrt{M\alpha})- 2S\) solution, where \( M\) is the Lipschitz constant of the Hessian matrix.

#### Main Contributions

1. **Simple and Efficient Algorithm**: The proposed algorithm is based on a simple line search method and evaluates second - order information (i.e., the noisy Hessian matrix) only when necessary, thus improving the computational efficiency.
2. **Practicality and Speed**: Compared with other differentially private algorithms (such as DP - TR), this algorithm has better practical performance, especially showing a significant speed advantage in high - dimensional problems.
3. **Acceleration Techniques**: By introducing line search and mini - batching, the convergence process of the algorithm is further accelerated and the required sample complexity is reduced.

In summary, this paper proposes a new method to efficiently solve the non - convex ERM problem while ensuring data privacy by combining differential privacy techniques and optimization algorithms.