Abstract:We present a series of new differentially private (DP) algorithms with dimension-independent margin guarantees. For the family of linear hypotheses, we give a pure DP learning algorithm that benefits from relative deviation margin guarantees, as well as an efficient DP learning algorithm with margin guarantees. We also present a new efficient DP learning algorithm with margin guarantees for kernel-based hypotheses with shift-invariant kernels, such as Gaussian kernels, and point out how our results can be extended to other kernels using oblivious sketching techniques. We further give a pure DP learning algorithm for a family of feed-forward neural networks for which we prove margin guarantees that are independent of the input dimension. Additionally, we describe a general label DP learning algorithm, which benefits from relative deviation margin bounds and is applicable to a broad family of hypothesis sets, including that of neural networks. Finally, we show how our DP learning algorithms can be augmented in a general way to include model selection, to select the best confidence margin parameter.
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is: how to design learning algorithms with margin guarantees for common hypothesis sets (such as linear hypotheses, kernel - based hypotheses, and neural networks, etc.) while maintaining differential privacy (DP). Specifically, the authors hope to overcome the dimension - dependence problem in existing differentially private PAC learning and empirical risk minimization (ERM) methods, thereby providing learning guarantees independent of the input dimension.
### Core problems of the paper
1. **Favorable learning guarantees under differential privacy**:
- Can favorable (dimension - independent) differential privacy learning be achieved in standard hypothesis sets?
- Must it rely on distribution - related margins?
2. **Margin guarantees in classification**:
- In classification problems, traditional learning margins based on dimension or VC dimension are too pessimistic. Therefore, the authors hope to obtain more useful learning guarantees through margin - based learning bounds.
3. **Margin analysis without strong assumptions**:
- Existing research usually relies on the hard - margin separability assumption, which often does not hold in practical applications. Therefore, the authors hope to conduct margin analysis without relying on such strong assumptions.
### Main contributions
1. **Differential privacy algorithms on linear hypothesis sets**:
- A pure differential privacy (pure DP) learning algorithm is proposed, which has relative deviation margin guarantees but is computationally inefficient.
- An efficient differential privacy learning algorithm with margin guarantees is proposed, and its margin is independent of the input dimension.
2. **Differential privacy algorithms on kernel - based hypothesis sets**:
- A new efficient differential privacy learning algorithm is proposed, which is suitable for hypothesis sets using shift - invariant kernels (such as Gaussian kernels) and can be extended to other types of kernel functions (such as polynomial kernels and neural tangent kernels NTK).
3. **Differential privacy learning on neural networks**:
- A pure differential privacy learning algorithm suitable for feed - forward neural networks is designed, and it is proved that its margin guarantees are independent of the input dimension and have a good dependence on network parameters.
4. **Label privacy learning algorithms**:
- A label privacy learning algorithm is proposed, which is suitable for a wide range of hypothesis sets, including neural networks, and benefits from relative deviation margin guarantees.
5. **Model selection**:
- It is shown how to generally enhance differential privacy learning algorithms to include model selection, so as to select the best confidence margin parameters.
### Markdown representation of formulas
- **Margin loss**:
\[
R_\rho^D(h)=\mathbb{E}_{z =(x,y)\sim D}[1_{yh(x)\leq\rho}]
\]
\[
\hat{R}_\rho^S(h)=\mathbb{E}_{z =(x,y)\sim S}[1_{yh(x)\leq\rho}]
\]
- **ρ - hinge loss**:
\[
\ell_\rho(u):=\max(1 - u/\rho,0),\quad u\in\mathbb{R}
\]
\[
L_\rho^D(w)=\mathbb{E}_{z=(x,y)\sim D}[\ell_\rho(y\langle w,x\rangle)]
\]
\[
\hat{L}_\rho^S(w)=\mathbb{E}_{z=(x,y)\sim S}[\ell_\rho(y\langle w,x\rangle)]
\]
- **Definition of differential privacy**:
\[
P(A(S)\i