Abstract:The vulnerability of machine learning models in adversarial scenarios has garnered significant interest in the academic community over the past decade, resulting in a myriad of attacks and defenses. However, while the community appears to be overtly successful in devising new attacks across new contexts, the development of defenses has stalled. After a decade of research, we appear no closer to securing AI applications beyond additional training. Despite a lack of effective mitigations, AI development and its incorporation into existing systems charge full speed ahead with the rise of generative AI and large language models. Will our ineffectiveness in developing solutions to adversarial threats further extend to these new technologies? In this paper, we argue that overly permissive attack and overly restrictive defensive threat models have hampered defense development in the ML domain. Through the lens of adversarial evasion attacks against neural networks, we critically examine common attack assumptions, such as the ability to bypass any defense not explicitly built into the model. We argue that these flawed assumptions, seen as reasonable by the community based on paper acceptance, have encouraged the development of adversarial attacks that map poorly to real-world scenarios. In turn, new defenses evaluated against these very attacks are inadvertently required to be almost perfect and incorporated as part of the model. But do they need to? In practice, machine learning models are deployed as a small component of a larger system. We analyze adversarial machine learning from a system security perspective rather than an AI perspective and its implications for emerging AI paradigms.

Lights Toward Adversarial Machine Learning: The Achilles Heel of Artificial Intelligence

Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain

Defense Against Adversarial Ai

The Adversarial Machine Learning Conundrum: Can the Insecurity of ML Become the Achilles' Heel of Cognitive Networks?

Adversarial Machine Learning and Cybersecurity: Risks, Challenges, and Legal Implications

Adversarial Machine Learning -- Industry Perspectives

Security and Privacy for Artificial Intelligence: Opportunities and Challenges

Adversarial Machine Learning for Cybersecurity and Computer Vision: Current Developments and Challenges

Law and Adversarial Machine Learning

Adversarial Attacks in Machine Learning: Key Insights and Defense Approaches

Taking off the Rose-Tinted Glasses: A Critical Look at Adversarial ML Through the Lens of Evasion Attacks

Security and Machine Learning in the Real World

Evaluating the Vulnerabilities in ML systems in terms of adversarial attacks

Mental Models of Adversarial Machine Learning

Politics of Adversarial Machine Learning

Can We Rely on AI?

Techniques for Adversarial Examples Threatening the Safety of Artificial Intelligence Based Systems

Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning

Ten AI Stepping Stones for Cybersecurity

Security Matters: A Survey on Adversarial Machine Learning