Abstract:Machine learning models are known to be susceptible to adversarial perturbation. One famous attack is the adversarial patch, a sticker with a particularly crafted pattern that makes the model incorrectly predict the object it is placed on. This attack presents a critical threat to cyber-physical systems that rely on cameras such as autonomous cars. Despite the significance of the problem, conducting research in this setting has been difficult; evaluating attacks and defenses in the real world is exceptionally costly while synthetic data are unrealistic. In this work, we propose the REAP (REalistic Adversarial Patch) benchmark, a digital benchmark that allows the user to evaluate patch attacks on real images, and under real-world conditions. Built on top of the Mapillary Vistas dataset, our benchmark contains over 14,000 traffic signs. Each sign is augmented with a pair of geometric and lighting transformations, which can be used to apply a digitally generated patch realistically onto the sign. Using our benchmark, we perform the first large-scale assessments of adversarial patch attacks under realistic conditions. Our experiments suggest that adversarial patch attacks may present a smaller threat than previously believed and that the success rate of an attack on simpler digital simulations is not predictive of its actual effectiveness in practice. We release our benchmark publicly at <a class="link-external link-https" href="https://github.com/wagner-group/reap-benchmark" rel="external noopener nofollow">this https URL</a>.
Computer Vision and Pattern Recognition,Artificial Intelligence,Cryptography and Security,Machine Learning
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the vulnerability of machine - learning models to adversarial perturbations, especially adversarial patch attacks. Specifically, an adversarial patch is a carefully - designed sticker that, when placed on a target object, can cause a machine - learning model to misidentify the object. Such attacks pose a serious threat to camera - dependent cyber - physical systems, such as self - driving cars. For example, a malicious actor can place a sticker on a stop sign, causing a self - driving car to mistake it for a speed - limit sign and thus not stop.
Although this problem is very important, research progress has been limited because it is very difficult to quantitatively assess this threat. Experiments in the real world are costly and difficult to carry out on a large scale, while evaluations using synthetic data lack realism. Therefore, this paper proposes REAP (Realistic Adversarial Patch Benchmark), a digital benchmarking platform for evaluating adversarial patch attacks. REAP is built based on the Mapillary Vistas dataset and contains more than 14,000 traffic - sign images. Through geometric and illumination transformations, it can realistically apply digitally - generated patches to these signs.
### Main contributions of the paper
1. **Large - scale evaluation**: REAP contains 14,651 traffic - sign images, which enables researchers to quantitatively analyze the effectiveness of attacks and defenses.
2. **Realistic patch rendering**: REAP provides a set of tools that can realistically render any digital patch onto a traffic sign, taking into account factors such as the patch's position, camera angle, and illumination conditions. Moreover, these transformations are fast and differentiable, allowing back - propagation through the rendering process.
3. **Realistic image distribution**: The images in REAP are taken under real - world conditions, including different sizes, distances, illumination conditions, and degrees of occlusion.
### Experimental results
1. **Limited effectiveness of existing attacks**: The success rate of existing adversarial patch attacks on the REAP benchmark is not high and they are unable to succeed on most images. In contrast, the success rate of simple attack models (such as ℓp - bounded perturbations or patch attacks on simple benchmarks) is close to 100%. Moreover, adversarially - trained models can almost completely prevent these attacks while having little impact on the performance of benign data.
2. **Performance on synthetic data does not reflect performance on REAP**: The study found that the correlation between the success rate of attacks on the synthetic version of the benchmark and the full REAP is poor, indicating that under more realistic conditions, the performance of the synthetic benchmark cannot predict the actual effectiveness of attacks.
3. **Illumination and patch position are particularly important**: During the patch - rendering process, illumination transformations and the patch's position have the greatest impact on the attack success rate. In contrast, perspective transformations, although also important, have a relatively smaller impact on the attack success rate.
### Conclusion
The REAP benchmark provides a more accurate and realistic evaluation platform for research on adversarial patch attacks. Through REAP, researchers can more accurately evaluate new attack and defense methods, thus promoting further development in this field.