Abstract:We propose an efficient solver for the privacy funnel (PF) method, leveraging its difference-of-convex (DC) structure. The proposed DC separation results in a closed-form update equation, which allows straightforward application to both known and unknown distribution settings. For known distribution case, we prove the convergence (local stationary points) of the proposed non-greedy solver, and empirically show that it outperforms the state-of-the-art approaches in characterizing the privacy-utility trade-off. The insights of our DC approach apply to unknown distribution settings where labeled empirical samples are available instead. Leveraging the insights, our alternating minimization solver satisfies the fundamental Markov relation of PF in contrast to previous variational inference-based solvers. Empirically, we evaluate the proposed solver with MNIST and Fashion-MNIST datasets. Our results show that under a comparable reconstruction quality, an adversary suffers from higher prediction error from clustering our compressed codes than that with the compared methods. Most importantly, our solver is independent to private information in inference phase contrary to the baselines.
What problem does this paper attempt to address?
This paper attempts to solve several key problems in the Privacy Funnel (PF) method, specifically including:
1. **Non - convex optimization problem**:
The core of the Privacy Funnel method is a non - convex optimization problem, with the goal of minimizing the trade - off between privacy leakage and information utility. Specifically, the PF method aims to solve the following optimization problem:
\[
\min_{P(Z|X) \in \Omega_{Z|X}} I(Z;Y), \quad \text{subject to} \quad I(Z;X) \geq \eta_X
\]
where $\eta_X$ is a control threshold, and $I(Z;X)$ and $I(Z;Y)$ represent the mutual information between random variables $Z$ and $X$, and between $Z$ and $Y$ respectively. Due to the non - convexity of this problem, it is very difficult to solve.
2. **Strict Markov constraint**:
The optimization problem is subject to a strict Markov chain constraint $Y \rightarrow X \rightarrow Z$, which further increases the complexity of the solution.
3. **Requirement for complete knowledge of the joint distribution**:
Traditional methods require complete knowledge of the joint probability distribution $P(X, Y)$, but in practical applications, obtaining the complete joint distribution is usually unrealistic. Therefore, how to effectively solve the problem with only empirical samples is an important issue.
4. **Limitations of existing methods**:
Existing solution methods usually simplify the problem by restricting the feasible solution set, for example, using greedy algorithms or variational inference methods, but these methods have problems such as slow convergence speed and inability to satisfy Markov relationships, thus affecting the ability to characterize the privacy - utility trade - off.
### Main contributions of the paper
To solve the above problems, the author proposes an effective solver based on the Difference - of - Convex (DC) structure. The main contributions are as follows:
- **Closed - form update equations**: Using the DC structure, closed - form update equations are derived, enabling the solver to be applied to known and unknown distribution settings.
- **Convergence and performance improvement**: For the known distribution case, the local convergence of the proposed non - greedy solver is proved, and experiments show that it is superior to existing methods in characterizing the privacy - utility trade - off.
- **Applicable to unknown distributions**: In the case of unknown distributions, empirical samples are used for training to ensure that the solver conforms to the basic Markov relationship of PF, so that it does not need to rely on private information in the inference stage.
- **Deep learning implementation**: The solver is combined with a deep neural network and applied to the MNIST and Fashion - MNIST datasets. The experimental results show that under the same reconstruction quality, the prediction error obtained by the adversary from clustering the compressed codes is higher, verifying the effectiveness of the method.
In summary, this paper significantly improves the ability of the Privacy Funnel method to characterize the privacy - utility trade - off by introducing the DC structure and an efficient solution strategy, especially when dealing with unknown distributions and large - scale datasets.