Abstract:Despite considerable efforts on making them robust, real-world ML-based systems remain vulnerable to decision based attacks, as definitive proofs of their operational robustness have so far proven intractable. The canonical approach in robustness evaluation calls for adaptive attacks, that is with complete knowledge of the defense and tailored to bypass it. In this study, we introduce a more expansive notion of being adaptive and show how attacks but also defenses can benefit by it and by learning from each other through interaction. We propose and evaluate a framework for adaptively optimizing black-box attacks and defenses against each other through the competitive game they form. To reliably measure robustness, it is important to evaluate against realistic and worst-case attacks. We thus augment both attacks and the evasive arsenal at their disposal through adaptive control, and observe that the same can be done for defenses, before we evaluate them first apart and then jointly under a multi-agent perspective. We demonstrate that active defenses, which control how the system responds, are a necessary complement to model hardening when facing decision-based attacks; then how these defenses can be circumvented by adaptive attacks, only to finally elicit active and adaptive defenses. We validate our observations through a wide theoretical and empirical investigation to confirm that AI-enabled adversaries pose a considerable threat to black-box ML-based systems, rekindling the proverbial arms race where defenses have to be AI-enabled too. Succinctly, we address the challenges posed by adaptive adversaries and develop adaptive defenses, thereby laying out effective strategies in ensuring the robustness of ML-based systems deployed in the real-world.

A Game Theoretic Perspective on Adversarial Machine Learning and Related Cybersecurity Applications

A Survey of Game Theoretic Approaches for Adversarial Machine Learning in Cybersecurity Tasks

A Game Model for Adversarial Classification in Spam Filtering

Adversarial Machine Learning for Cybersecurity and Computer Vision: Current Developments and Challenges

Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain

Taxonomy and Recent Advance of Game Theoretical Approaches in Adversarial Machine Learning: A Survey

Adversarial Markov Games: On Adaptive Decision-Based Attacks and Defenses

Simulations of Event-Based Cyber Dynamics via Adversarial Machine Learning

Game-Theoretic Design of Secure and Resilient Distributed Support Vector Machines with Adversaries

Defenses in Adversarial Machine Learning: A Survey

Adversarial Machine Learning: Bayesian Perspectives

Game Theoretic Mixed Experts for Combinational Adversarial Machine Learning

The Threat of Adversarial Attacks on Machine Learning in Network Security -- A Survey

Adversarial for Good – Defending Training Data Privacy with Adversarial Attack Wisdom

A Markov Game Model for AI-based Cyber Security Attack Mitigation

Security Matters: A Survey on Adversarial Machine Learning

Motivating the Rules of the Game for Adversarial Example Research

Randomized Prediction Games for Adversarial Machine Learning

A Game Theoretic Analysis of Additive Adversarial Attacks and Defenses

The Space of Adversarial Strategies