Abstract:We present a Distributionally Robust Optimization (DRO) approach to estimate a robustified regression plane in a linear regression setting, when the observed samples are potentially contaminated with adversarially corrupted outliers. Our approach mitigates the impact of outliers by hedging against a family of probability distributions on the observed data, some of which assign very low probabilities to the outliers. The set of distributions under consideration are close to the empirical distribution in the sense of the Wasserstein metric. We show that this DRO formulation can be relaxed to a convex optimization problem which encompasses a class of models. By selecting proper norm spaces for the Wasserstein metric, we are able to recover several commonly used regularized regression models. We provide new insights into the regularization term and give guidance on the selection of the regularization coefficient from the standpoint of a confidence region. We establish two types of performance guarantees for the solution to our formulation under mild conditions. One is related to its out-of-sample behavior (prediction bias), and the other concerns the discrepancy between the estimated and true regression planes (estimation bias). Extensive numerical results demonstrate the superiority of our approach to a host of regression models, in terms of the prediction and estimation accuracies. We also consider the application of our robust learning procedure to outlier detection, and show that our approach achieves a much higher AUC (Area Under the ROC Curve) than M-estimation (Huber, 1964, 1973).

An integrated perspective of robustness in regression through the lens of the bias-variance trade-off

Robust Regularized Kernel Regression.

Homotopy continuation approaches for robust SV classification and regression

Sparse and Robust Linear Regression: An Optimization Algorithm and Its Statistical Properties

A Fundamental Accuracy--Robustness Trade-off in Regression and Classification

Bayesian Robustness to Outliers in Linear Regression and Ratio Estimation

A Robust Learning Approach for Regression Models Based on Distributionally Robust Optimization.

A NEW PERSPECTIVE ON ROBUST M-ESTIMATION: FINITE SAMPLE THEORY AND APPLICATIONS TO DEPENDENCE-ADJUSTED MULTIPLE TESTING

The Price of Implicit Bias in Adversarially Robust Generalization

Universal Robust Regression via Maximum Mean Discrepancy

COMPARISONS BETWEEN ROBUST REGRESSION APPROACHES IN THE PRESENCE OF OUTLIERS AND HIGH LEVERAGE POINTS

Variable-Sized Uncertainty and Inverse Problems in Robust Optimization

On the Adversarial Robustness of Robust Estimators

The Price of Robustness

On Restricting Influence of Anomalous Data on Predicted Values of a Regression Model

Outlier Robust and Sparse Estimation of Linear Regression Coefficients

A unified approach to inverse robust optimization problems

Sensitivity of robust optimization problems under drift and volatility uncertainty

A Robust Learning Algorithm for Regression Models Using Distributionally Robust Optimization under the Wasserstein Metric

Robust Regression over Averaged Uncertainty

Unified robust estimation