Adversarial machine learning at scale. arXiv 2016

Alexey Kurakin, Ian Goodfellow, Samy Bengio

2016-01-01

Abstract:

What problem does this paper attempt to address?

Adversarial Machine Learning at Scale

Alexey Kurakin,Ian Goodfellow,Samy Bengio

DOI: https://doi.org/10.48550/arXiv.1611.01236

2016-11-04

Computer Vision and Pattern Recognition

Abstract:Adversarial examples are malicious inputs designed to fool machine learning models. They often transfer from one model to another, allowing attackers to mount black box attacks without knowledge of the target model's parameters. Adversarial training is the process of explicitly training a model on adversarial examples, in order to make it more robust to attack or to reduce its test error on clean inputs. So far, adversarial training has primarily been applied to small problems. In this research, we apply adversarial training to ImageNet. Our contributions include: (1) recommendations for how to succesfully scale adversarial training to large models and datasets, (2) the observation that adversarial training confers robustness to single-step attack methods, (3) the finding that multi-step attack methods are somewhat less transferable than single-step attack methods, so single-step attacks are the best for mounting black-box attacks, and (4) resolution of a "label leaking" effect that causes adversarially trained models to perform better on adversarial examples than on clean examples, because the adversarial example construction process uses the true label and the model can learn to exploit regularities in the construction process.
Revisiting Adversarial Training at Scale

Zeyu Wang,Xianhang Li,Hongru Zhu,Cihang Xie

2024-04-21

Abstract:The machine learning community has witnessed a drastic change in the training pipeline, pivoted by those ''foundation models'' with unprecedented scales. However, the field of adversarial training is lagging behind, predominantly centered around small model sizes like ResNet-50, and tiny and low-resolution datasets like CIFAR-10. To bridge this transformation gap, this paper provides a modern re-examination with adversarial training, investigating its potential benefits when applied at scale. Additionally, we introduce an efficient and effective training strategy to enable adversarial training with giant models and web-scale data at an affordable computing cost. We denote this newly introduced framework as AdvXL.

Computer Vision and Pattern Recognition
Adversarial Machine Learning: Bayesian Perspectives

David Rios Insua,Roi Naveiro,Victor Gallego,Jason Poulos

DOI: https://doi.org/10.1080/01621459.2023.2183129

2024-02-22

Abstract:Adversarial Machine Learning (AML) is emerging as a major field aimed at protecting machine learning (ML) systems against security threats: in certain scenarios there may be adversaries that actively manipulate input data to fool learning systems. This creates a new class of security vulnerabilities that ML systems may face, and a new desirable property called adversarial robustness essential to trust operations based on ML outputs. Most work in AML is built upon a game-theoretic modelling of the conflict between a learning system and an adversary, ready to manipulate input data. This assumes that each agent knows their opponent's interests and uncertainty judgments, facilitating inferences based on Nash equilibria. However, such common knowledge assumption is not realistic in the security scenarios typical of AML. After reviewing such game-theoretic approaches, we discuss the benefits that Bayesian perspectives provide when defending ML-based systems. We demonstrate how the Bayesian approach allows us to explicitly model our uncertainty about the opponent's beliefs and interests, relaxing unrealistic assumptions, and providing more robust inferences. We illustrate this approach in supervised learning settings, and identify relevant future research problems.

Artificial Intelligence,Machine Learning,Computation
Trustworthy Machine Learning under Social and Adversarial Data Sources

Han Shao

2024-08-03

Abstract:Machine learning has witnessed remarkable breakthroughs in recent years. As machine learning permeates various aspects of daily life, individuals and organizations increasingly interact with these systems, exhibiting a wide range of social and adversarial behaviors. These behaviors may have a notable impact on the behavior and performance of machine learning systems. Specifically, during these interactions, data may be generated by strategic individuals, collected by self-interested data collectors, possibly poisoned by adversarial attackers, and used to create predictors, models, and policies satisfying multiple objectives. As a result, the machine learning systems' outputs might degrade, such as the susceptibility of deep neural networks to adversarial examples (Shafahi et al., 2018; Szegedy et al., 2013) and the diminished performance of classic algorithms in the presence of strategic individuals (Ahmadi et al., 2021). Addressing these challenges is imperative for the success of machine learning in societal settings.

Machine Learning,Artificial Intelligence,Computer Science and Game Theory
Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain

Ishai Rosenberg,Asaf Shabtai,Yuval Elovici,Lior Rokach

DOI: https://doi.org/10.1145/3453158

IF: 16.6

2022-06-30

ACM Computing Surveys

Abstract:In recent years, machine learning algorithms, and more specifically deep learning algorithms, have been widely used in many fields, including cyber security. However, machine learning systems are vulnerable to adversarial attacks, and this limits the application of machine learning, especially in non-stationary, adversarial environments, such as the cyber security domain, where actual adversaries (e.g., malware developers) exist. This article comprehensively summarizes the latest research on adversarial attacks against security solutions based on machine learning techniques and illuminates the risks they pose. First, the adversarial attack methods are characterized based on their stage of occurrence, and the attacker’ s goals and capabilities. Then, we categorize the applications of adversarial attack and defense methods in the cyber security domain. Finally, we highlight some characteristics identified in recent research and discuss the impact of recent advancements in other adversarial learning domains on future research directions in the cyber security domain. To the best of our knowledge, this work is the first to discuss the unique challenges of implementing end-to-end adversarial attacks in the cyber security domain, map them in a unified taxonomy, and use the taxonomy to highlight future research directions.

computer science, theory & methods
A Game Theoretic Perspective on Adversarial Machine Learning and Related Cybersecurity Applications

Yan Zhou,Murat Kantarcioglu,Bowei Xi

DOI: https://doi.org/10.1002/9781119723950.ch13

2021-09-12

Abstract:In cybersecurity applications where machine learning algorithms are increasingly used to detect vulnerabilities, a somewhat unique challenge arises as exploits targeting machine learning models are constantly devised by the attackers. Traditional machine learning models are no longer robust and reliable when they are under attack. The action and reaction between machine learning systems and the adversary can be modeled as a game between two or more players. Under well‐defined attack models, game theory can provide robustness guarantee for machine learning models that are otherwise vulnerable to application‐time data corruption. We review two cases of game theory‐based machine learning techniques: in one case, players play a zero sum game by following a minimax strategy, while in the other case, players play a sequential game with one player as the leader and the rest as the followers. Experimental results on e‐mail spam and web spam datasets are presented. In the zero sum game, we demonstrate that an adversarial SVM model built upon the minimax strategy is much more resilient to adversarial attacks than standard SVM and one‐class SVM models. We also show that optimal learning strategies derived to counter overly pessimistic attack models can produce unsatisfactory results when the real attacks are much weaker. In the sequential game, we demonstrate that the mixed strategy, allowing a player to randomize over available strategies, is the best solution in general without knowing what types of adversaries machine learning applications are facing in the wild. We also discuss scenarios where players' behavior may derail rational decision making and models that consider such decision risks.
Law and Adversarial Machine Learning

Ram Shankar Siva Kumar,David R. O'Brien,Kendra Albert,Salome Vilojen

DOI: https://doi.org/10.48550/arXiv.1810.10731

2018-12-05

Abstract:When machine learning systems fail because of adversarial manipulation, how should society expect the law to respond? Through scenarios grounded in adversarial ML literature, we explore how some aspects of computer crime, copyright, and tort law interface with perturbation, poisoning, model stealing and model inversion attacks to show how some attacks are more likely to result in liability than others. We end with a call for action to ML researchers to invest in transparent benchmarks of attacks and defenses; architect ML systems with forensics in mind and finally, think more about adversarial machine learning in the context of civil liberties. The paper is targeted towards ML researchers who have no legal background.

Machine Learning,Cryptography and Security,Computers and Society
Scaling Laws for Black box Adversarial Attacks

Chuan Liu,Huanran Chen,Yichi Zhang,Yinpeng Dong,Jun Zhu

2024-11-25

Abstract:A longstanding problem of deep learning models is their vulnerability to adversarial examples, which are often generated by applying imperceptible perturbations to natural examples. Adversarial examples exhibit cross-model transferability, enabling to attack black-box models with limited information about their architectures and parameters. Model ensembling is an effective strategy to improve the transferability by attacking multiple surrogate models simultaneously. However, as prior studies usually adopt few models in the ensemble, there remains an open question of whether scaling the number of models can further improve black-box attacks. Inspired by the findings in large foundation models, we investigate the scaling laws of black-box adversarial attacks in this work. By analyzing the relationship between the number of surrogate models and transferability of adversarial examples, we conclude with clear scaling laws, emphasizing the potential of using more surrogate models to enhance adversarial transferability. Extensive experiments verify the claims on standard image classifiers, multimodal large language models, and even proprietary models like GPT-4o, demonstrating consistent scaling effects and impressive attack success rates with more surrogate models. Further studies by visualization indicate that scaled attacks bring better interpretability in semantics, indicating that the common features of models are captured.

Machine Learning,Computer Vision and Pattern Recognition
Scaling Compute Is Not All You Need for Adversarial Robustness

Edoardo Debenedetti,Zishen Wan,Maksym Andriushchenko,Vikash Sehwag,Kshitij Bhardwaj,Bhavya Kailkhura

2023-12-20

Abstract:The last six years have witnessed significant progress in adversarially robust deep learning. As evidenced by the CIFAR-10 dataset category in RobustBench benchmark, the accuracy under $\ell_\infty$ adversarial perturbations improved from 44\% in \citet{Madry2018Towards} to 71\% in \citet{peng2023robust}. Although impressive, existing state-of-the-art is still far from satisfactory. It is further observed that best-performing models are often very large models adversarially trained by industrial labs with significant computational budgets. In this paper, we aim to understand: ``how much longer can computing power drive adversarial robustness advances?" To answer this question, we derive \emph{scaling laws for adversarial robustness} which can be extrapolated in the future to provide an estimate of how much cost we would need to pay to reach a desired level of robustness. We show that increasing the FLOPs needed for adversarial training does not bring as much advantage as it does for standard training in terms of performance improvements. Moreover, we find that some of the top-performing techniques are difficult to exactly reproduce, suggesting that they are not robust enough for minor changes in the training setup. Our analysis also uncovers potentially worthwhile directions to pursue in future research. Finally, we make our benchmarking framework (built on top of \texttt{timm}~\citep{rw2019timm}) publicly available to facilitate future analysis in efficient robust deep learning.

Machine Learning,Artificial Intelligence,Cryptography and Security
Adversarial Attacks on Machine Learning Systems for High-Frequency Trading

Micah Goldblum,Avi Schwarzschild,Ankit B. Patel,Tom Goldstein

DOI: https://doi.org/10.1145/3490354.3494367

2021-10-30

Abstract:Algorithmic trading systems are often completely automated, and deep learning is increasingly receiving attention in this domain. Nonetheless, little is known about the robustness properties of these models. We study valuation models for algorithmic trading from the perspective of adversarial machine learning. We introduce new attacks specific to this domain with size constraints that minimize attack costs. We further discuss how these attacks can be used as an analysis tool to study and evaluate the robustness properties of financial models. Finally, we investigate the feasibility of realistic adversarial attacks in which an adversarial trader fools automated trading systems into making inaccurate predictions.

Machine Learning,Cryptography and Security,Statistical Finance
Adversarial Attacks on Multi-Agent Communication

James Tu,Tsunhsuan Wang,Jingkang Wang,Sivabalan Manivasagam,Mengye Ren,Raquel Urtasun

DOI: https://doi.org/10.1109/iccv48922.2021.00767

2021-01-01

Abstract:Growing at a fast pace, modern autonomous systems will soon be deployed at scale, opening up the possibility for cooperative multi-agent systems. Sharing information and distributing workloads allow autonomous agents to better perform tasks and increase computation efficiency. However, shared information can be modified to execute adversarial attacks on deep learning models that are widely employed in modern systems. Thus, we aim to study the robustness of such systems and focus on exploring adversarial attacks in a novel multi-agent setting where communication is done through sharing learned intermediate representations of neural networks. We observe that an indistinguishable adversarial message can severely degrade performance, but becomes weaker as the number of benign agents increases. Furthermore, we show that black-box transfer attacks are more difficult in this setting when compared to directly perturbing the inputs, as it is necessary to align the distribution of learned representations with domain adaptation. Our work studies robustness at the neural network level to contribute an additional layer of fault tolerance to modern security protocols for more secure multi-agent systems.
Adversarial Learning in Real-World Fraud Detection: Challenges and Perspectives

Danele Lunghi,Alkis Simitsis,Olivier Caelen,Gianluca Bontempi

DOI: https://doi.org/10.1145/3600046.3600051

2023-07-04

Abstract:Data economy relies on data-driven systems and complex machine learning applications are fueled by them. Unfortunately, however, machine learning models are exposed to fraudulent activities and adversarial attacks, which threaten their security and trustworthiness. In the last decade or so, the research interest on adversarial machine learning has grown significantly, revealing how learning applications could be severely impacted by effective attacks. Although early results of adversarial machine learning indicate the huge potential of the approach to specific domains such as image processing, still there is a gap in both the research literature and practice regarding how to generalize adversarial techniques in other domains and applications. Fraud detection is a critical defense mechanism for data economy, as it is for other applications as well, which poses several challenges for machine learning. In this work, we describe how attacks against fraud detection systems differ from other applications of adversarial machine learning, and propose a number of interesting directions to bridge this gap.

Machine Learning,Cryptography and Security
Adversarial Machine Learning for Cybersecurity and Computer Vision: Current Developments and Challenges

Bowei Xi

DOI: https://doi.org/10.48550/arXiv.2107.02894

2021-06-30

Abstract:We provide a comprehensive overview of adversarial machine learning focusing on two application domains, i.e., cybersecurity and computer vision. Research in adversarial machine learning addresses a significant threat to the wide application of machine learning techniques -- they are vulnerable to carefully crafted attacks from malicious adversaries. For example, deep neural networks fail to correctly classify adversarial images, which are generated by adding imperceptible perturbations to clean <a class="link-external link-http" href="http://images.We" rel="external noopener nofollow">this http URL</a> first discuss three main categories of attacks against machine learning techniques -- poisoning attacks, evasion attacks, and privacy attacks. Then the corresponding defense approaches are introduced along with the weakness and limitations of the existing defense approaches. We notice adversarial samples in cybersecurity and computer vision are fundamentally different. While adversarial samples in cybersecurity often have different properties/distributions compared with training data, adversarial images in computer vision are created with minor input perturbations. This further complicates the development of robust learning techniques, because a robust learning technique must withstand different types of attacks.

Cryptography and Security,Machine Learning
Mental Models of Adversarial Machine Learning

Lukas Bieringer,Kathrin Grosse,Michael Backes,Battista Biggio,Katharina Krombholz

DOI: https://doi.org/10.48550/arXiv.2105.03726

2022-06-29

Abstract:Although machine learning is widely used in practice, little is known about practitioners' understanding of potential security challenges. In this work, we close this substantial gap and contribute a qualitative study focusing on developers' mental models of the machine learning pipeline and potentially vulnerable components. Similar studies have helped in other security fields to discover root causes or improve risk communication. Our study reveals two \facets of practitioners' mental models of machine learning security. Firstly, practitioners often confuse machine learning security with threats and defences that are not directly related to machine learning. Secondly, in contrast to most academic research, our participants perceive security of machine learning as not solely related to individual models, but rather in the context of entire workflows that consist of multiple components. Jointly with our additional findings, these two facets provide a foundation to substantiate mental models for machine learning security and have implications for the integration of adversarial machine learning into corporate workflows, \new{decreasing practitioners' reported uncertainty}, and appropriate regulatory frameworks for machine learning security.

Cryptography and Security,Artificial Intelligence
Adversarial Machine Learning -- Industry Perspectives

Ram Shankar Siva Kumar,Magnus Nyström,John Lambert,Andrew Marshall,Mario Goertzel,Andi Comissoneru,Matt Swann,Sharon Xia

DOI: https://doi.org/10.48550/arXiv.2002.05646

2021-03-20

Abstract:Based on interviews with 28 organizations, we found that industry practitioners are not equipped with tactical and strategic tools to protect, detect and respond to attacks on their Machine Learning (ML) systems. We leverage the insights from the interviews and we enumerate the gaps in perspective in securing machine learning systems when viewed in the context of traditional software security development. We write this paper from the perspective of two personas: developers/ML engineers and security incident responders who are tasked with securing ML systems as they are designed, developed and deployed ML systems. The goal of this paper is to engage researchers to revise and amend the Security Development Lifecycle for industrial-grade software in the adversarial ML era.

Computers and Society,Cryptography and Security,Machine Learning
Towards Model-Agnostic Adversarial Defenses using Adversarially Trained Autoencoders

Pratik Vaishnavi,Kevin Eykholt,Atul Prakash,Amir Rahmati

DOI: https://doi.org/10.48550/arXiv.1909.05921

2020-03-30

Abstract:Adversarial machine learning is a well-studied field of research where an adversary causes predictable errors in a machine learning algorithm through precise manipulation of the input. Numerous techniques have been proposed to harden machine learning algorithms and mitigate the effect of adversarial attacks. Of these techniques, adversarial training, which augments the training data with adversarial samples, has proven to be an effective defense with respect to a certain class of attacks. However, adversarial training is computationally expensive and its improvements are limited to a single model. In this work, we take a first step toward creating a model-agnostic adversarial defense. We propose Adversarially-Trained Autoencoder Augmentation (AAA), the first model-agnostic adversarial defense that is robust against certain adaptive adversaries. We show that AAA allows us to achieve a partially model-agnostic defense by training a single autoencoder to protect multiple pre-trained classifiers; achieving adversarial performance on par or better than adversarial training without modifying the classifiers. Furthermore, we demonstrate that AAA can be used to create a fully model-agnostic defense for MNIST and Fashion MNIST datasets by improving the adversarial performance of a never before seen pre-trained classifier by at least 45% with no additional training. Finally, using a natural image corruption dataset, we show that our approach improves robustness to naturally corrupted images,which has been identified as strongly indicative of true adversarial robustness.

Computer Vision and Pattern Recognition,Machine Learning
Adversarial Machine Learning on Social Network: A Survey

Sensen Guo,Xiaoyu Li,Zhiying Mu

DOI: https://doi.org/10.3389/fphy.2021.766540

IF: 3.718

2021-11-29

Frontiers in Physics

Abstract:In recent years, machine learning technology has made great improvements in social networks applications such as social network recommendation systems, sentiment analysis, and text generation. However, it cannot be ignored that machine learning algorithms are vulnerable to adversarial examples, that is, adding perturbations that are imperceptible to the human eye to the original data can cause machine learning algorithms to make wrong outputs with high probability. This also restricts the widespread use of machine learning algorithms in real life. In this paper, we focus on adversarial machine learning algorithms on social networks in recent years from three aspects: sentiment analysis, recommendation system, and spam detection, We review some typical applications of machine learning algorithms and adversarial example generation and defense algorithms for machine learning algorithms in the above three aspects in recent years. besides, we also analyze the current research progress and prospects for the directions of future research.

physics, multidisciplinary
Adversarial Machine Learning for Social Good: Reframing the Adversary as an Ally

Shawqi Al-Maliki,Adnan Qayyum,Hassan Ali,Mohamed Abdallah,Junaid Qadir,Dinh Thai Hoang,Dusit Niyato,Ala Al-Fuqaha

2023-10-05

Abstract:Deep Neural Networks (DNNs) have been the driving force behind many of the recent advances in machine learning. However, research has shown that DNNs are vulnerable to adversarial examples -- input samples that have been perturbed to force DNN-based models to make errors. As a result, Adversarial Machine Learning (AdvML) has gained a lot of attention, and researchers have investigated these vulnerabilities in various settings and modalities. In addition, DNNs have also been found to incorporate embedded bias and often produce unexplainable predictions, which can result in anti-social AI applications. The emergence of new AI technologies that leverage Large Language Models (LLMs), such as ChatGPT and GPT-4, increases the risk of producing anti-social applications at scale. AdvML for Social Good (AdvML4G) is an emerging field that repurposes the AdvML bug to invent pro-social applications. Regulators, practitioners, and researchers should collaborate to encourage the development of pro-social applications and hinder the development of anti-social ones. In this work, we provide the first comprehensive review of the emerging field of AdvML4G. This paper encompasses a taxonomy that highlights the emergence of AdvML4G, a discussion of the differences and similarities between AdvML4G and AdvML, a taxonomy covering social good-related concepts and aspects, an exploration of the motivations behind the emergence of AdvML4G at the intersection of ML4G and AdvML, and an extensive summary of the works that utilize AdvML4G as an auxiliary tool for innovating pro-social applications. Finally, we elaborate upon various challenges and open research issues that require significant attention from the research community.

Machine Learning,Computers and Society
Learning Images Across Scales Using Adversarial Training

Krzysztof Wolski,Adarsh Djeacoumar,Alireza Javanmardi,Hans-Peter Seidel,Christian Theobalt,Guillaume Cordonnier,Karol Myszkowski,George Drettakis,Xingang Pan,Thomas Leimkühler

DOI: https://doi.org/10.1145/3658190

IF: 6.2

2024-07-19

ACM Transactions on Graphics

Abstract:The real world exhibits rich structure and detail across many scales of observation. It is difficult, however, to capture and represent a broad spectrum of scales using ordinary images. We devise a novel paradigm for learning a representation that captures an orders-of-magnitude variety of scales from an unstructured collection of ordinary images. We treat this collection as a distribution of scale-space slices to be learned using adversarial training, and additionally enforce coherency across slices. Our approach relies on a multiscale generator with carefully injected procedural frequency content, which allows to interactively explore the emerging continuous scale space. Training across vastly different scales poses challenges regarding stability, which we tackle using a supervision scheme that involves careful sampling of scales. We show that our generator can be used as a multiscale generative model, and for reconstructions of scale spaces from unstructured patches. Significantly outperforming the state of the art, we demonstrate zoom-in factors of up to 256x at high quality and scale consistency.

computer science, software engineering

Adversarial machine learning at scale. arXiv 2016

Adversarial Machine Learning at Scale

Revisiting Adversarial Training at Scale

Adversarial Machine Learning: Bayesian Perspectives

Trustworthy Machine Learning under Social and Adversarial Data Sources

Adversarial Machine Learning Attacks and Defense Methods in the Cyber Security Domain

A Game Theoretic Perspective on Adversarial Machine Learning and Related Cybersecurity Applications

Law and Adversarial Machine Learning

Scaling Laws for Black box Adversarial Attacks

Scaling Compute Is Not All You Need for Adversarial Robustness

Adversarial Attacks on Machine Learning Systems for High-Frequency Trading

Adversarial Attacks on Multi-Agent Communication

Adversarial Learning in Real-World Fraud Detection: Challenges and Perspectives

Adversarial Machine Learning for Cybersecurity and Computer Vision: Current Developments and Challenges

Mental Models of Adversarial Machine Learning

Adversarial Machine Learning -- Industry Perspectives

Towards Model-Agnostic Adversarial Defenses using Adversarially Trained Autoencoders

Adversarial Machine Learning on Social Network: A Survey

Adversarial Machine Learning for Social Good: Reframing the Adversary as an Ally

Learning Images Across Scales Using Adversarial Training