Abstract:State-of-the-art machine learning models can be vulnerable to very small input perturbations that are adversarially constructed. Adversarial training is an effective approach to defend against it. Formulated as a min-max problem, it searches for the best solution when the training data were corrupted by the worst-case attacks. Linear models are among the simple models where vulnerabilities can be observed and are the focus of our study. In this case, adversarial training leads to a convex optimization problem which can be formulated as the minimization of a finite sum. We provide a comparative analysis between the solution of adversarial training in linear regression and other regularization methods. Our main findings are that: (A) Adversarial training yields the minimum-norm interpolating solution in the overparameterized regime (more parameters than data), as long as the maximum disturbance radius is smaller than a threshold. And, conversely, the minimum-norm interpolator is the solution to adversarial training with a given radius. (B) Adversarial training can be equivalent to parameter shrinking methods (ridge regression and Lasso). This happens in the underparametrized region, for an appropriate choice of adversarial radius and zero-mean symmetrically distributed covariates. (C) For $\ell_\infty$-adversarial training -- as in square-root Lasso -- the choice of adversarial radius for optimal bounds does not depend on the additive noise variance. We confirm our theoretical findings with numerical examples.

Lepskii Principle in Supervised Learning

Balancing Principle in Supervised Learning for a General Regularization Scheme

Kernel ridge vs. principal component regression: minimax bounds and adaptability of regularization operators

Regularization in kernel learning

Kernel regression, minimax rates and effective dimensionality: beyond the regular case

Generalization errors of Laplacian regularized least squares regression

Robust Regularized Kernel Regression.

Analysis of Least Squares Regularized Regression in Reproducing Kernel Krein Spaces

Lepskii Principle for Distributed Kernel Ridge Regression

Optimal learning rates for Kernel Conjugate Gradient regression

Online Kernel Learning with a Near Optimal Sparsity Bound

Learning Rates of Least-Square Regularized Regression with Polynomial Kernels

Learning Theory Estimates for Coefficient-Based Regularized Regression

Analysis of Regularized Least-Squares in Reproducing Kernel Kreĭn Spaces

Early stopping and polynomial smoothing in regression with reproducing kernels

Convergence Conditions of Online Regularized Statistical Learning in Reproducing Kernel Hilbert Space With Non-Stationary Data

Regularization properties of adversarially-trained linear regression

Kernel ridge vs. principal component regression: Minimax bounds and the qualification of regularization operators

Enhanced Feature Learning via Regularisation: Integrating Neural Networks and Kernel Methods

A Note on Adaptive Lp Regularization

Kernel Partial Least Squares is Universally Consistent