Scaling Compute Is Not All You Need for Adversarial Robustness

Edoardo Debenedetti,Zishen Wan,Maksym Andriushchenko,Vikash Sehwag,Kshitij Bhardwaj,Bhavya Kailkhura

2023-12-20

Abstract:The last six years have witnessed significant progress in adversarially robust deep learning. As evidenced by the CIFAR-10 dataset category in RobustBench benchmark, the accuracy under $\ell_\infty$ adversarial perturbations improved from 44\% in \citet{Madry2018Towards} to 71\% in \citet{peng2023robust}. Although impressive, existing state-of-the-art is still far from satisfactory. It is further observed that best-performing models are often very large models adversarially trained by industrial labs with significant computational budgets. In this paper, we aim to understand: ``how much longer can computing power drive adversarial robustness advances?" To answer this question, we derive \emph{scaling laws for adversarial robustness} which can be extrapolated in the future to provide an estimate of how much cost we would need to pay to reach a desired level of robustness. We show that increasing the FLOPs needed for adversarial training does not bring as much advantage as it does for standard training in terms of performance improvements. Moreover, we find that some of the top-performing techniques are difficult to exactly reproduce, suggesting that they are not robust enough for minor changes in the training setup. Our analysis also uncovers potentially worthwhile directions to pursue in future research. Finally, we make our benchmarking framework (built on top of \texttt{timm}~\citep{rw2019timm}) publicly available to facilitate future analysis in efficient robust deep learning.

Machine Learning,Artificial Intelligence,Cryptography and Security

What problem does this paper attempt to address?

This paper explores whether simply increasing computational resources is a feasible solution for adversarial robustness. In recent years, although deep learning has made progress in adversarial robustness, there are still significant deficiencies. The research indicates that the current state-of-the-art models are often large models trained adversarially by industrial labs with abundant computational budgets. The paper systematically trains numerous models on the CIFAR-10 dataset and analyzes the impact of factors such as model size, adversarial loss, and attack steps on robustness.

The results show that increasing the floating point operations (FLOPs) for adversarial training does not bring the same performance improvement as standard training. Moreover, some top techniques are difficult to replicate precisely, suggesting their lack of stability with minor variations in the training settings. The paper also reveals future research directions worth exploring and provides a benchmark framework to facilitate analysis in efficient and robust deep learning.

The paper points out that relying solely on computational power to enhance adversarial robustness may be inefficient or even infeasible. This is not only because increasing computational resources directly affects the environment, but also because it may result in disproportionate environmental costs without correspondingly improving adversarial robustness. Therefore, the paper calls for the community to go beyond the traditional paradigm of simply increasing computational resources and seek innovative approaches to address this challenge, in order to achieve more robust and reliable AI systems.