Abstract:Advances in deep learning have led to the broad adoption of Deep Neural Networks (DNNs) to a range of important machine learning problems, e.g., guiding autonomous vehicles, speech recognition, malware detection. Yet, machine learning models, including DNNs, were shown to be vulnerable to adversarial samples-subtly (and often humanly indistinguishably) modified malicious inputs crafted to compromise the integrity of their outputs. Adversarial examples thus enable adversaries to manipulate system behaviors. Potential attacks include attempts to control the behavior of vehicles, have spam content identified as legitimate content, or have malware identified as legitimate software. Adversarial examples are known to transfer from one model to another, even if the second model has a different architecture or was trained on a different set. We introduce the first practical demonstration that this cross-model transfer phenomenon enables attackers to control a remotely hosted DNN with no access to the model, its parameters, or its training data. In our demonstration, we only assume that the adversary can observe outputs from the target DNN given inputs chosen by the adversary. We introduce the attack strategy of fitting a substitute model to the input-output pairs in this manner, then crafting adversarial examples based on this auxiliary model. We evaluate the approach on existing DNN datasets and real-world settings. In one experiment, we force a DNN supported by MetaMind (one of the online APIs for DNN classifiers) to mis-classify inputs at a rate of 84.24%. We conclude with experiments exploring why adversarial samples transfer between DNNs, and a discussion on the applicability of our attack when targeting machine learning algorithms distinct from DNNs.

Adversarial Examples in Physical World

Adversarial Example Attacks in Internet of Things (IoT)

Attack As Defense: Characterizing Adversarial Examples Using Robustness.

Adversarial examples: attacks and defenses in the physical world

Adversarial Examples in the Physical World: A Survey

Adversarial Examples: Attacks and Defenses for Deep Learning

Physical World Adversarial Attacks on Images and Texts

Adversarial Examples: Opportunities and Challenges

Adversarial Attack? Don't Panic

Robust Physical-World Attacks on Deep Learning Models

Contextual Adversarial Attack Against Aerial Detection in the Physical World

Adversarial examples in the physical world

A Survey on Physical Adversarial Attack in Computer Vision

Physical Adversarial Attack meets Computer Vision: A Decade Survey

Adversarial Examples on Object Recognition: A Comprehensive Survey

Adversarial Examples in Deep Learning: Characterization and Divergence

Visually Adversarial Attacks and Defenses in the Physical World: A Survey

Physical Adversarial Attacks on an Aerial Imagery Object Detector

Research Progress and Challenges on Application-Driven Adversarial Examples: A Survey

Practical Black-Box Attacks against Deep Learning Systems using Adversarial Examples