Abstract:Ensemble defenses, are widely employed in various security-related applications to enhance model performance and robustness. The widespread adoption of these techniques also raises many questions: Are general ensembles defenses guaranteed to be more robust than individuals? Will stronger adaptive attacks defeat existing ensemble defense strategies as the cybersecurity arms race progresses? Can ensemble defenses achieve adversarial robustness to different types of attacks simultaneously and resist the continually adjusted adaptive attacks? Unfortunately, these critical questions remain unresolved as there are no platforms for comprehensive evaluation of ensemble adversarial attacks and defenses in the cybersecurity domain. In this paper, we propose a general Cybersecurity Adversarial Robustness Evaluation (CARE) platform aiming to bridge this gap.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the following three key problems: 1. **Are model ensembles and ensemble defenses necessarily more robust than a single model?** - Existing research has insufficiently evaluated the adversarial robustness of ensemble models, especially in the field of network security. Therefore, it is not clear whether ensemble models and ensemble defenses can ensure greater robustness than a single model. 2. **Will stronger adaptive attacks defeat existing ensemble defense strategies?** - As the adversarial in the field of network security continues to escalate, attackers will continuously adjust their attack methods to adapt to the defense system. Therefore, it is necessary to evaluate the effectiveness of existing ensemble defense strategies when facing stronger adaptive attacks. 3. **Can ensemble defenses resist multiple types of attacks simultaneously and withstand continuously - adjusted adaptive attacks?** - Existing defense methods (such as naive adversarial training) are usually optimized for only one type of attack, which makes them vulnerable to other types of attacks. Therefore, it is necessary to explore whether ensemble defense methods can handle multiple attacks simultaneously and resist adaptive attacks. To solve these problems, the authors propose a general Cybersecurity Adversarial Robustness Evaluation platform (CARE), which has the following features: - **Unified interface**: Provide a unified interface to evaluate heterogeneous ensemble models and transfer ensemble attacks in classical machine learning and deep learning. - **Comprehensive evaluation**: Support the comprehensive evaluation of ML/DL vulnerabilities in security applications using multiple gradient and gradient - free methods. - **Practical application**: Be able to implement more realistic adaptive attacks and consider many problem - space constraints in the security domain to ensure the generation of real adversarial samples. - **Scalability**: Be easy to expand to incorporate the latest attack/defense methods and new problem - space constraints. Through this platform, the authors hope to systematically evaluate the robustness of ensemble models and ensemble defenses in security applications, thereby answering the above - mentioned key questions. The experimental results show that the existing model ensembles and ensemble adversarial training methods cannot ensure the enhancement of the model's robustness, while the robust ensemble adversarial training they proposed can resist multiple attack methods and adaptive attacks simultaneously. ### Formula representation To ensure the correctness and readability of the formulas, the following are some important formulas involved in the paper: - Adversarial sample generation: \[ \hat{x} = x+\delta, \quad \text{where } \delta=\arg\min_{\delta' \in S} l(x, x + \delta') \] where \( l(x, x+\delta') \) represents the distance function between feature vectors, and \( S \) is the perturbation set. - Loss function: \[ L(f(\hat{x}; \theta), y) \] where \( f(\hat{x}; \theta) \) is the output of the target model, \( y \) is the true label, and \( \theta \) is the model parameter. - Attack objective function: \[ G_a(\hat{x})=\arg\max_{\hat{x}} L(f(\hat{x}; \theta), y) \] - Defense objective function: \[ G_d(\hat{x})=\arg\min_{\hat{x}} L(f(\hat{x}; \theta), y) \] These formulas are used to describe the basic principles of adversarial attacks and defenses.

CARE: Ensemble Adversarial Robustness Evaluation Against Adaptive Attackers for Security Applications

Attack As Defense: Characterizing Adversarial Examples Using Robustness.

How Effective Are the Prevailing Attack-Defense Models for Cybersecurity Anyway?

Boosting Adversarial Training in Safety-Critical Systems Through Boundary Data Selection

Adversarial Robust Decision-Making under Uncertainty Learning and Dynamic Ensemble Selection

Certifying Joint Adversarial Robustness for Model Ensembles

Dynamic Defense Approach for Adversarial Robustness in Deep Neural Networks via Stochastic Ensemble Smoothed Model

Synergy-of-Experts: Collaborate to Improve Adversarial Robustness

Omni: Automated Ensemble with Unexpected Models against Adversarial Evasion Attack

Understanding and Improving Ensemble Adversarial Defense

Voting based ensemble improves robustness of defensive models

CMMR: A Composite Multidimensional Models Robustness Evaluation Framework for Deep Learning.

Practical Evaluation of Adversarial Robustness Via Adaptive Auto Attack.

DEEPSEC: A Uniform Platform for Security Analysis of Deep Learning Model.

Ensemble Methods as a Defense to Adversarial Perturbations Against Deep Neural Networks

Ensemble-in-One: Ensemble Learning Within Random Gated Networks for Enhanced Adversarial Robustness.

Improving Adversarial Robustness via Promoting Ensemble Diversity.

Dynamic ensemble selection based on Deep Neural Network Uncertainty Estimation for Adversarial Robustness

Improving Adversarial Robustness Via Promoting Ensemble Diversity

Diversity supporting robustness: Enhancing adversarial robustness via differentiated ensemble predictions

DetectS Ec: Evaluating the Robustness of Object Detection Models to Adversarial Attacks