Abstract:Our goal is to improve reliability of Machine Learning (ML) systems deployed in the wild. ML models perform exceedingly well when test examples are similar to train examples. However, real-world applications are required to perform on any distribution of test examples. Current ML systems can fail silently on test examples with distribution shifts. In order to improve reliability of ML models due to covariate or domain shift, we propose algorithms that enable models to: (a) generalize to a larger family of test distributions, (b) evaluate accuracy under distribution shifts, (c) adapt to a target distribution. We study causes of impaired robustness to domain shifts and present algorithms for training domain robust models. A key source of model brittleness is due to domain overfitting, which our new training algorithms suppress and instead encourage domain-general hypotheses. While we improve robustness over standard training methods for certain problem settings, performance of ML systems can still vary drastically with domain shifts. It is crucial for developers and stakeholders to understand model vulnerabilities and operational ranges of input, which could be assessed on the fly during the deployment, albeit at a great cost. Instead, we advocate for proactively estimating accuracy surfaces over any combination of prespecified and interpretable domain shifts for performance forecasting. We present a label-efficient estimation to address estimation over a combinatorial space of domain shifts. Further, when a model's performance on a target domain is found to be poor, traditional approaches adapt the model using the target domain's resources. Standard adaptation methods assume access to sufficient labeled resources, which may be impractical for deployed models. We initiate a study of lightweight adaptation techniques with only unlabeled data resources with a focus on language applications.

Methods for Deep Learning Model Failure Detection and Model Adaption: A Survey

Beyond Model Adaptation at Test Time: A Survey

Detecting Defects in Deep Learning Systems: a Survey

Deep Model Based Domain Adaptation for Fault Diagnosis.

A Failure Prediction Approach Based on BiLSTM and Deep Feature Extractor for Hard Disk Drives.

A Survey of Deep Graph Learning under Distribution Shifts: from Graph Out-of-Distribution Generalization to Adaptation

Robust Computer Vision in an Ever-Changing World: A Survey of Techniques for Tackling Distribution Shifts

Monitoring Machine Learning Models: Online Detection of Relevant Deviations

Beyond Generalization: A Survey of Out-Of-Distribution Adaptation on Graphs

Active Model Adaptation Under Unknown Shift

Towards Reliable Learning in the Wild: Generalization and Adaptation

Distributed Data Center Failure Analysis Model Based on Deep Learning.

Robustness, Evaluation and Adaptation of Machine Learning Models in the Wild

A new method for intelligent fault diagnosis of machines based on unsupervised domain adaptation

System-level Hardware Failure Prediction Using Deep Learning

A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts

A study on adaptation lightweight architecture based deep learning models for bearing fault diagnosis under varying working conditions

Theoretically Guaranteed Distribution Adaptable Learning

Graph Learning under Distribution Shifts: A Comprehensive Survey on Domain Adaptation, Out-of-distribution, and Continual Learning

Navigating Distribution Shifts in Medical Image Analysis: A Survey

Online Data Drift Detection for Anomaly Detection Services based on Deep Learning towards Multivariate Time Series