Abstract:We study the problem of learning under arbitrary distribution shift, where the learner is trained on a labeled set from one distribution but evaluated on a different, potentially adversarially generated test distribution. We focus on two frameworks: PQ learning [Goldwasser, A. Kalai, Y. Kalai, Montasser NeurIPS 2020], allowing abstention on adversarially generated parts of the test distribution, and TDS learning [Klivans, Stavropoulos, Vasilyan COLT 2024], permitting abstention on the entire test distribution if distribution shift is detected. All prior known algorithms either rely on learning primitives that are computationally hard even for simple function classes, or end up abstaining entirely even in the presence of a tiny amount of distribution shift. We address both these challenges for natural function classes, including intersections of halfspaces and decision trees, and standard training distributions, including Gaussians. For PQ learning, we give efficient learning algorithms, while for TDS learning, our algorithms can tolerate moderate amounts of distribution shift. At the core of our approach is an improved analysis of spectral outlier-removal techniques from learning with nasty noise. Our analysis can (1) handle arbitrarily large fraction of outliers, which is crucial for handling arbitrary distribution shifts, and (2) obtain stronger bounds on polynomial moments of the distribution after outlier removal, yielding new insights into polynomial regression under distribution shifts. Lastly, our techniques lead to novel results for tolerant testable learning [Rubinfeld and Vasilyan STOC 2023], and learning with nasty noise.

Tolerant Algorithms for Learning with Arbitrary Covariate Shift

Efficient Discrepancy Testing for Learning with Distribution Shift

Learning Intersections of Halfspaces with Distribution Shift: Improved Algorithms and SQ Lower Bounds

Testable Learning with Distribution Shift

Beyond Invariance: Test-Time Label-Shift Adaptation for Distributions with "Spurious" Correlations

Stable Adversarial Learning under Distributional Shifts

Learning a Single Neuron Robustly to Distributional Shifts and Adversarial Label Noise

Blessings and Curses of Covariate Shifts: Adversarial Learning Dynamics, Directional Convergence, and Equilibria

Why stable learning works? a theory of covariate shift generalization

Adaptive conformal inference under distribution shift

Unleashing the power of graph data augmentation on covariate distribution shift

Enhancing Distributional Stability among Sub-populations

Out of the Ordinary: Spectrally Adapting Regression for Covariate Shift

Training Classifiers under Covariate Shift by Constructing the Maximum Consistent Distribution Subset

Biquality learning: a framework to design algorithms dealing with closed-set distribution shifts

Algorithmic Fairness Generalization under Covariate and Dependence Shifts Simultaneously

Transformation-Invariant Learning and Theoretical Guarantees for OOD Generalization

Beyond Discrepancy: A Closer Look at the Theory of Distribution Shift

Certifiable Out-of-Distribution Generalization.

Conformal Inference for Online Prediction with Arbitrary Distribution Shifts