Adversarial examples in the physical world

Alexey Kurakin,Ian Goodfellow,Samy Bengio

DOI: https://doi.org/10.48550/arXiv.1607.02533

2017-02-11

Abstract:Most existing machine learning classifiers are highly vulnerable to adversarial examples. An adversarial example is a sample of input data which has been modified very slightly in a way that is intended to cause a machine learning classifier to misclassify it. In many cases, these modifications can be so subtle that a human observer does not even notice the modification at all, yet the classifier still makes a mistake. Adversarial examples pose security concerns because they could be used to perform an attack on machine learning systems, even if the adversary has no access to the underlying model. Up to now, all previous work have assumed a threat model in which the adversary can feed data directly into the machine learning classifier. This is not always the case for systems operating in the physical world, for example those which are using signals from cameras and other sensors as an input. This paper shows that even in such physical world scenarios, machine learning systems are vulnerable to adversarial examples. We demonstrate this by feeding adversarial images obtained from cell-phone camera to an ImageNet Inception classifier and measuring the classification accuracy of the system. We find that a large fraction of adversarial examples are classified incorrectly even when perceived through the camera.

Computer Vision and Pattern Recognition,Cryptography and Security,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the vulnerability of machine - learning systems to adversarial examples in the real world. Specifically, the paper explores whether these adversarial examples can still lead to classification errors when they are input to machine - learning classifiers through sensors such as cameras. Before this, most studies assumed that attackers could directly provide input data to machine - learning models, but this assumption does not hold in many real - world application scenarios, such as robots perceiving the world through cameras, video surveillance systems, image or sound classification in mobile applications, etc. Therefore, this paper aims to verify whether adversarial examples generated in the physical world can successfully attack machine - learning systems, even if these samples need to be collected through devices such as cameras. This not only involves the effectiveness of adversarial examples, but also relates to the security of machine - learning systems in the real world.