Abstract:Model Inversion (MI) attacks aim at leveraging the output information of target models to reconstruct privacy-sensitive training data, raising widespread concerns on privacy threats of Deep Neural Networks (DNNs). Unfortunately, in tandem with the rapid evolution of MI attacks, the lack of a comprehensive, aligned, and reliable benchmark has emerged as a formidable challenge. This deficiency leads to inadequate comparisons between different attack methods and inconsistent experimental setups. In this paper, we introduce the first practical benchmark for model inversion attacks and defenses to address this critical gap, which is named \textit{MIBench}. This benchmark serves as an extensible and reproducible modular-based toolbox and currently integrates a total of 16 state-of-the-art attack and defense methods. Moreover, we furnish a suite of assessment tools encompassing 9 commonly used evaluation protocols to facilitate standardized and fair evaluation and analysis. Capitalizing on this foundation, we conduct extensive experiments from multiple perspectives to holistically compare and analyze the performance of various methods across different scenarios, which overcomes the misalignment issues and discrepancy prevalent in previous works. Based on the collected attack methods and defense strategies, we analyze the impact of target resolution, defense robustness, model predictive power, model architectures, transferability and loss function. Our hope is that this \textit{MIBench} could provide a unified, practical and extensible toolbox and is widely utilized by researchers in the field to rigorously test and compare their novel methods, ensuring equitable evaluations and thereby propelling further advancements in the future development.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: **Lack of a comprehensive, fair, and reliable model inversion attack (MI) and defense benchmarking framework**. Specifically, with the rapid development of MI attacks, existing research work is often limited to comparison with a small number of previous works when evaluating new methods, resulting in a limited scope and depth of analysis. In addition, the experimental settings are inconsistent among different studies, and there are also differences in evaluation metrics. These problems greatly reduce the effectiveness and fairness of comparative studies.
To address these challenges, the authors proposed **MIBench**, which is the first comprehensive benchmarking toolbox for model inversion attacks and defenses. MIBench aims to provide an extensible and reproducible modular framework, integrating 16 state - of - the - art attack and defense methods and providing 9 commonly - used evaluation protocols. Through this benchmark, researchers can more comprehensively and fairly evaluate and compare different MI attack and defense methods, thereby promoting the further development of this field.
### Main contributions of the paper:
1. **Constructed the first comprehensive MI benchmarking framework**, as an extensible and reproducible modular toolbox, helping researchers more conveniently evaluate the effectiveness of MI attacks.
2. **Implemented 16 latest attack methods and defense strategies**, as well as 9 evaluation protocols, further identifying the most effective defense measures to deal with potential MI attacks.
3. **Conducted extensive experiments**, evaluating different MI methods from multiple perspectives and studying the influence of different factors, providing new insights for the MI field. In particular, it verified the association between stronger model prediction ability and higher privacy leakage risk.
### Solutions to key problems:
- **Unified experimental protocol**: MIBench provides standardized experimental settings and evaluation metrics, ensuring fair comparison between different methods.
- **Multi - dimensional evaluation**: Through multiple evaluation metrics (such as Accuracy, Feature Distance, FID, etc.), comprehensively measure the effects of MI attacks and defenses.
- **Cross - resolution evaluation**: It not only covers low - resolution images but also extends to high - resolution images, reflecting the complexity in practical applications.
In summary, MIBench fills the gap in benchmarking in the MI field, providing researchers with a unified, practical, and extensible toolbox, which is helpful for promoting the further development of MI attack and defense technologies.