Abstract:Learning-based pattern classifiers, including deep networks, have shown impressive performance in several application domains, ranging from computer vision to cybersecurity. However, it has also been shown that adversarial input perturbations carefully crafted either at training or at test time can easily subvert their predictions. The vulnerability of machine learning to such wild patterns (also referred to as adversarial examples), along with the design of suitable countermeasures, have been investigated in the research field of adversarial machine learning. In this work, we provide a thorough overview of the evolution of this research area over the last ten years and beyond, starting from pioneering, earlier work on the security of non-deep learning algorithms up to more recent work aimed to understand the security properties of deep learning algorithms, in the context of computer vision and cybersecurity tasks. We report interesting connections between these apparently-different lines of work, highlighting common misconceptions related to the security evaluation of machine-learning algorithms. We review the main threat models and attacks defined to this end, and discuss the main limitations of current work, along with the corresponding future challenges towards the design of more secure learning algorithms.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the security and robustness issues of machine - learning models when facing adversarial inputs (i.e., input samples carefully designed to mislead model predictions, also known as adversarial samples). Specifically, the paper reviews the evolution of the field of adversarial machine learning over the past decade, from early research on the security of non - deep - learning algorithms to recent research on the security of deep - learning algorithms, especially in applications in computer vision and network security tasks. The paper explores the research background of adversarial machine learning, main threat models and attack methods, the limitations of current research, and future challenges, aiming to promote the design of more secure learning algorithms. ### Main Problems 1. **Vulnerability of Adversarial Samples**: The paper points out that even deep - learning models with excellent performance are vulnerable to adversarial inputs, which can cause model prediction errors through small and imperceptible perturbations. 2. **Development History of Adversarial Machine Learning**: The paper emphasizes that adversarial machine learning did not start in 2014, but there were related studies as early as 2004. Early work mainly focused on the security of spam filtering and biometric systems. 3. **Threat Models and Attack Methods**: The paper discusses in detail how to model an adversary's knowledge, capabilities, and goals, and proposes methods for evaluating the security of learning algorithms. 4. **Defense Mechanisms**: The paper explores existing defense mechanisms and their limitations, and proposes directions for future research to design more secure machine - learning algorithms. ### Key Points - **Definition of Adversarial Samples**: Adversarial samples refer to samples that cause machine - learning models to make incorrect predictions by slightly perturbing input data. - **Threat Models**: Including an adversary's knowledge (such as whether they know the training data, feature set, learning algorithm, etc.), capabilities (such as whether they can manipulate the training data or test data), and goals (such as integrity attacks, availability attacks, privacy attacks, etc.). - **Attack Strategies**: Define attack strategies in the form of optimization problems, such as maximizing the test error or minimizing the classification accuracy. - **Security Evaluation Curve**: Evaluate the performance degradation of learning algorithms by increasing the attack intensity, thereby providing a more comprehensive security evaluation. ### Goals The goal of the paper is to connect the achievements of different research fields, clarify common misunderstandings about machine - learning security evaluation, and propose challenges and directions for future research to promote the development of more secure machine - learning algorithms.

Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning

Adversarial Machine Learning for Cybersecurity and Computer Vision: Current Developments and Challenges

Security Matters: A Survey on Adversarial Machine Learning

Adversarial examples: A survey of attacks and defenses in deep learning-enabled cybersecurity systems

Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain

Security and Machine Learning in the Real World

Adversarial Examples on Object Recognition: A Comprehensive Survey

Adversarial Examples: Attacks and Defenses for Deep Learning

Defenses in Adversarial Machine Learning: A Survey

Practical Attacks on Machine Learning: A Case Study on Adversarial Windows Malware

Adversarial Examples in Deep Learning: Characterization and Divergence

Adversarial Attacks in Machine Learning: Key Insights and Defense Approaches

Adversarial Examples: Opportunities and Challenges

Adversarial machine learning in Network Intrusion Detection Systems

Data Driven Exploratory Attacks on Black Box Classifiers in Adversarial Domains

Research Progress and Challenges on Application-Driven Adversarial Examples: A Survey

Lights Toward Adversarial Machine Learning: The Achilles Heel of Artificial Intelligence

Adversarial Attacks and Defences: A Survey

A survey of robust adversarial training in pattern recognition: Fundamental, theory, and methodologies