Abstract:Membership Inference Attacks have emerged as a dominant method for empirically measuring privacy leakage from machine learning models. Here, privacy is measured by the {\em{advantage}} or gap between a score or a function computed on the training and the test data. A major barrier to the practical deployment of these attacks is that they do not scale to large well-generalized models -- either the advantage is relatively low, or the attack involves training multiple models which is highly compute-intensive. In this work, inspired by discrepancy theory, we propose a new empirical privacy metric that is an upper bound on the advantage of a family of membership inference attacks. We show that this metric does not involve training multiple models, can be applied to large Imagenet classification models in-the-wild, and has higher advantage than existing metrics on models trained with more recent and sophisticated training recipes. Motivated by our empirical results, we also propose new membership inference attacks tailored to these training losses.
What problem does this paper attempt to address?
### What problem does this paper attempt to solve?
This paper aims to solve the problem of privacy leakage measurement in **Membership Inference Attacks (MIA)** in machine - learning models. Specifically, the authors focus on how to evaluate the privacy - protection ability of machine - learning models more accurately and efficiently, especially when dealing with large - scale, well - generalized models.
#### Background and Challenges
1. **Limitations of existing methods**:
- **Score - based MIA**: These methods measure privacy leakage by calculating the score difference of the model on training data and test data. Although they are computationally efficient, they perform poorly on large - scale, well - generalized models and have low advantages.
- **Shadow Model - based MIA**: This method improves the attack effect by training multiple shadow models, but the computational cost is extremely high and it is difficult to apply to large - scale models.
2. **Obstacles to practical deployment**:
- Existing methods cannot be effectively extended to large, well - generalized models because these models have low advantages or require a large amount of computational resources to train shadow models.
#### Core contributions of the paper
1. **Introduce new privacy metric indicators**:
- Inspired by **Discrepancy Theory**, the authors propose a new empirical privacy metric indicator, which is the upper bound of the advantage of specific - class membership inference attacks.
- Propose an easily - computable approximation **CPM (Convex Polytope Machine)** as an approximation of this upper bound.
2. **Verify the effectiveness of the new indicator**:
- Experiments show that CPM has a higher advantage than existing popular attack methods, and due to its lightweight computation, it can be applied to large - scale ImageNet classification models.
3. **Discover and improve MIA for modern models**:
- The research finds that for advanced models using complex training methods, the gap between CPM and existing attack methods is larger. This indicates that the existing score functions may over - fit the standard models, and it is necessary to design score functions more suitable for modern models.
- Based on this finding, the authors propose new MIAs for specific training methods, such as **MixUp score** and **RelaxLoss score**, to better adapt to these complex models.
### Formula summary
- **Definition of MIA advantage**:
\[
\text{Adv}(m; f, S, D) := P_{z \sim S}(m(z, f)=1)-P_{z \sim D}(m(z, f)=1)\in[- 1,1]
\]
- **Definition of discrete distance**:
\[
D_Q(S, D):=\sup_{Q\in Q}\vert P_{(x,y)\sim S}((f(x), y)\in Q)-P_{(x,y)\sim D}((f(x), y)\in Q)\vert
\]
- **CPM objective function**:
\[
\max_{w_i\in\mathbb{R}^{2C}, b_i\in\mathbb{R}, i\in[K], s = \pm1}\frac{1}{|S|}\sum_{z\sim S}\ell_{\text{lg}}\left(\max_{i\in[K]}w_i^{\top}a_{z,f}+b_i, s\right)+E_{z\sim D}\left[\ell_{\text{lg}}\left(\max_{i\in[K]}w_i^{\top}a_{z,f}+b_i, -s\right)\right]
\]
Through these contributions, the paper provides a more powerful and computationally feasible method to evaluate the privacy - protection ability of machine - learning models, especially when dealing with complex models.