Abstract:Despite considerable efforts on making them robust, real-world ML-based systems remain vulnerable to decision based attacks, as definitive proofs of their operational robustness have so far proven intractable. The canonical approach in robustness evaluation calls for adaptive attacks, that is with complete knowledge of the defense and tailored to bypass it. In this study, we introduce a more expansive notion of being adaptive and show how attacks but also defenses can benefit by it and by learning from each other through interaction. We propose and evaluate a framework for adaptively optimizing black-box attacks and defenses against each other through the competitive game they form. To reliably measure robustness, it is important to evaluate against realistic and worst-case attacks. We thus augment both attacks and the evasive arsenal at their disposal through adaptive control, and observe that the same can be done for defenses, before we evaluate them first apart and then jointly under a multi-agent perspective. We demonstrate that active defenses, which control how the system responds, are a necessary complement to model hardening when facing decision-based attacks; then how these defenses can be circumvented by adaptive attacks, only to finally elicit active and adaptive defenses. We validate our observations through a wide theoretical and empirical investigation to confirm that AI-enabled adversaries pose a considerable threat to black-box ML-based systems, rekindling the proverbial arms race where defenses have to be AI-enabled too. Succinctly, we address the challenges posed by adaptive adversaries and develop adaptive defenses, thereby laying out effective strategies in ensuring the robustness of ML-based systems deployed in the real-world.

Secure Learning and Mining in Adversarial Environments [Extended Abstract]

Adversarial for Good – Defending Training Data Privacy with Adversarial Attack Wisdom

Security Matters: A Survey on Adversarial Machine Learning

The security of machine learning in an adversarial setting: A survey

Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain

Secure Machine Learning, a Brief Overview

Defenses in Adversarial Machine Learning: A Survey

A Game Theoretic Perspective on Adversarial Machine Learning and Related Cybersecurity Applications

Adversarial Machine Learning for Cybersecurity and Computer Vision: Current Developments and Challenges

A Survey of Game Theoretic Approaches for Adversarial Machine Learning in Cybersecurity Tasks

Security and Machine Learning in the Real World

The Threat of Adversarial Attacks on Machine Learning in Network Security -- A Survey

Adversarial Machine Learning: Bayesian Perspectives

Adversarial Machine Learning Attacks against Intrusion Detection Systems: A Survey on Strategies and Defense

Adversarial Attacks in Machine Learning: Key Insights and Defense Approaches

Adversarial Markov Games: On Adaptive Decision-Based Attacks and Defenses

Adversarial Attacks and Defences: A Survey

Adversarial Learning in Real-World Fraud Detection: Challenges and Perspectives

A Survey on Adversarial Machine Learning for Code Data: Realistic Threats, Countermeasures, and Interpretations

Adversarial Attacks and Defenses in Machine Learning-Powered Networks: A Contemporary Survey