Abstract:Deep learning (DL) models are inherently vulnerable to adversarial examples - maliciously crafted inputs to trigger target DL models to misbehave - which significantly hinders the application of DL in security-sensitive domains. Intensive research on adversarial learning has led to an arms race between adversaries and defenders. Such plethora of emerging attacks and defenses raise many questions: Which attacks are more evasive, preprocessing-proof, or transferable? Which defenses are more effective, utility-preserving, or general? Are ensembles of multiple defenses more robust than individuals? Yet, due to the lack of platforms for comprehensive evaluation on adversarial attacks and defenses, these critical questions remain largely unsolved. In this paper, we present the design, implementation, and evaluation of DEEPSEC, a uniform platform that aims to bridge this gap. In its current implementation, DEEPSEC incorporates 16 state-of-the-art attacks with 10 attack utility metrics, and 13 state-of-the-art defenses with 5 defensive utility metrics. To our best knowledge, DEEPSEC is the first platform that enables researchers and practitioners to (i) measure the vulnerability of DL models, (ii) evaluate the effectiveness of various attacks/defenses, and (iii) conduct comparative studies on attacks/defenses in a comprehensive and informative manner. Leveraging DEEPSEC, we systematically evaluate the existing adversarial attack and defense methods, and draw a set of key findings, which demonstrate DEEPSEC's rich functionality, such as (1) the trade-off between misclassification and imperceptibility is empirically confirmed; (2) most defenses that claim to be universally applicable can only defend against limited types of attacks under restricted settings; (3) it is not necessary that adversarial examples with higher perturbation magnitude are easier to be detected; (4) the ensemble of multiple defenses cannot improve the overall defense capability, but can improve the lower bound of the defense effectiveness of individuals. Extensive analysis on DEEPSEC demonstrates its capabilities and advantages as a benchmark platform which can benefit future adversarial learning research.

DeepRobust: a Platform for Adversarial Attacks and Defenses

Attack As Defense: Characterizing Adversarial Examples Using Robustness.

DeepRobust: A PyTorch Library for Adversarial Attacks and Defenses

ROBY: Evaluating the adversarial robustness of a deep model by its decision boundaries

Deep Defense: Training DNNs with Improved Adversarial Robustness

DeepDefense: Training Deep Neural Networks with Improved Robustness.

DEEPSEC: A Uniform Platform for Security Analysis of Deep Learning Model.

Towards Deep Learning Models Resistant to Adversarial Attacks

Opportunities and Challenges in Deep Learning Adversarial Robustness: A Survey

A Comprehensive Evaluation Framework for Deep Model Robustness

Adversarial Robustness Toolbox v1.0.0

DetectS Ec: Evaluating the Robustness of Object Detection Models to Adversarial Attacks

Adversarial Attacks and Defenses in Deep Learning: From a Perspective of Cybersecurity

Adversarial Attacks on ML Defense Models Competition

$A^{3}D$: A Platform of Searching for Robust Neural Architectures and Efficient Adversarial Attacks

D etect S ec : Evaluating the robustness of object detection models to adversarial attacks

A Comprehensive Study of the Robustness for LiDAR-Based 3D Object Detectors Against Adversarial Attacks

Adversarial Examples: Attacks and Defenses for Deep Learning

ROBY: Evaluating the Robustness of a Deep Model by its Decision Boundaries

Adversarially-Aware Robust Object Detector.