Abstract:We study the error of linear regression in the face of adversarial attacks. In this framework, an adversary changes the input to the regression model in order to maximize the prediction error. We provide bounds on the prediction error in the presence of an adversary as a function of the parameter norm and the error in the absence of such an adversary. We show how these bounds make it possible to study the adversarial error using analysis from non-adversarial setups. The obtained results shed light on the robustness of overparameterized linear models to adversarial attacks. Adding features might be either a source of additional robustness or brittleness. On the one hand, we use asymptotic results to illustrate how double-descent curves can be obtained for the adversarial error. On the other hand, we derive conditions under which the adversarial error can grow to infinity as more features are added, while at the same time, the test error goes to zero. We show this behavior is caused by the fact that the norm of the parameter vector grows with the number of features. It is also established that $\ell_\infty$ and $\ell_2$-adversarial attacks might behave fundamentally differently due to how the $\ell_1$ and $\ell_2$-norms of random projections concentrate. We also show how our reformulation allows for solving adversarial training as a convex optimization problem. This fact is then exploited to establish similarities between adversarial training and parameter-shrinking methods and to study how the training might affect the robustness of the estimated models.

High-dimensional (Group) Adversarial Training in Linear Regression

Asymptotic Behavior of Adversarial Training Estimator under $\ell_\infty$-Perturbation

Overparameterized Linear Regression under Adversarial Attacks

Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionality

On the Geometry of Regularization in Adversarial Training: High-Dimensional Asymptotics and Generalization Bounds

Regularization properties of adversarially-trained linear regression

Adversarial Training with Generated Data in High-Dimensional Regression: An Asymptotic Study

Asymptotic Behavior of Adversarial Training in Binary Classification

A High Dimensional Statistical Model for Adversarial Training: Geometry and Trade-Offs

Surprises in adversarially-trained linear regression

Adversarial Vulnerability as a Consequence of On-Manifold Inseparibility

Stability Analysis and Generalization Bounds of Adversarial Training

Precise Tradeoffs in Adversarial Training for Linear Regression

Data-Dependent Stability Analysis of Adversarial Training

Generalization Properties of Adversarial Training for $\ell_0$-Bounded Adversarial Attacks

Adversarial Examples Exist in Two-Layer ReLU Networks for Low Dimensional Linear Subspaces

Convergence and Margin of Adversarial Training on Separable Data

Convergence of Adversarial Training in Overparametrized Neural Networks

Improving Adversarial Robustness of Deep Neural Networks Via Linear Programming

Generalization Bounds for Adversarial Contrastive Learning

Regularization for Adversarial Robust Learning