Abstract:Our goal is to improve reliability of Machine Learning (ML) systems deployed in the wild. ML models perform exceedingly well when test examples are similar to train examples. However, real-world applications are required to perform on any distribution of test examples. Current ML systems can fail silently on test examples with distribution shifts. In order to improve reliability of ML models due to covariate or domain shift, we propose algorithms that enable models to: (a) generalize to a larger family of test distributions, (b) evaluate accuracy under distribution shifts, (c) adapt to a target distribution. We study causes of impaired robustness to domain shifts and present algorithms for training domain robust models. A key source of model brittleness is due to domain overfitting, which our new training algorithms suppress and instead encourage domain-general hypotheses. While we improve robustness over standard training methods for certain problem settings, performance of ML systems can still vary drastically with domain shifts. It is crucial for developers and stakeholders to understand model vulnerabilities and operational ranges of input, which could be assessed on the fly during the deployment, albeit at a great cost. Instead, we advocate for proactively estimating accuracy surfaces over any combination of prespecified and interpretable domain shifts for performance forecasting. We present a label-efficient estimation to address estimation over a combinatorial space of domain shifts. Further, when a model's performance on a target domain is found to be poor, traditional approaches adapt the model using the target domain's resources. Standard adaptation methods assume access to sufficient labeled resources, which may be impractical for deployed models. We initiate a study of lightweight adaptation techniques with only unlabeled data resources with a focus on language applications.

Active Model Adaptation Under Unknown Shift

Adaptive Risk Minimization: Learning to Adapt to Domain Shift

Learning Distinctive Margin Toward Active Domain Adaptation

Anomaly Detection under Distribution Shift

Robustness, Evaluation and Adaptation of Machine Learning Models in the Wild

Model-Agnostic Random Weighting for Out-of-Distribution Generalization

Adapting Large Multimodal Models to Distribution Shifts: The Role of In-Context Learning

Adapting to Online Label Shift with Provable Guarantees

LOG: Active Model Adaptation for Label-Efficient OOD Generalization

Distributional Shift Adaptation using Domain-Specific Features

Learning to Adapt to Online Streams with Distribution Shifts

Online Continual Adaptation with Active Self-Training

Theory-inspired Label Shift Adaptation via Aligned Distribution Mixture

Improving the Generalization of Segmentation Foundation Model under Distribution Shift via Weakly Supervised Adaptation

ELSA: Efficient Label Shift Adaptation through the Lens of Semiparametric Models

Active learning with adaptive regularization

Robust Computer Vision in an Ever-Changing World: A Survey of Techniques for Tackling Distribution Shifts

A One-step Approach to Covariate Shift Adaptation

Adapting to Covariate Shift in Real-time by Encoding Trees with Motion Equations

Domain-wise Data Acquisition to Improve Performance under Distribution Shift

Coping with Label Shift via Distributionally Robust Optimisation