Abstract:Traditional machine learning paradigms are based on the assumption that both training and test data follow the same statistical pattern, which is mathematically referred to as Independent and Identically Distributed ($i.i.d.$). However, in real-world applications, this $i.i.d.$ assumption often fails to hold due to unforeseen distributional shifts, leading to considerable degradation in model performance upon deployment. This observed discrepancy indicates the significance of investigating the Out-of-Distribution (OOD) generalization problem. OOD generalization is an emerging topic of machine learning research that focuses on complex scenarios wherein the distributions of the test data differ from those of the training data. This paper represents the first comprehensive, systematic review of OOD generalization, encompassing a spectrum of aspects from problem definition, methodological development, and evaluation procedures, to the implications and future directions of the field. Our discussion begins with a precise, formal characterization of the OOD generalization problem. Following that, we categorize existing methodologies into three segments: unsupervised representation learning, supervised model learning, and optimization, according to their positions within the overarching learning process. We provide an in-depth discussion on representative methodologies for each category, further elucidating the theoretical links between them. Subsequently, we outline the prevailing benchmark datasets employed in OOD generalization studies. To conclude, we overview the existing body of work in this domain and suggest potential avenues for future research on OOD generalization. A summary of the OOD generalization methodologies surveyed in this paper can be accessed at <a class="link-external link-http" href="http://out-of-distribution-generalization.com" rel="external noopener nofollow">this http URL</a>.

Data Distribution Transfer for Out Of Distribution Generalization

DecAug: Out-of-Distribution Generalization Via Decomposed Feature Representation and Semantic Augmentation

Towards Robust Out-of-Distribution Generalization: Data Augmentation and Neural Architecture Search Approaches

Deep Stable Learning for Out-Of-Distribution Generalization

OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization

Distribution Free Domain Generalization.

Certifiable Out-of-Distribution Generalization.

Out-of-Distribution (OOD) Detection and Generalization Improved by Augmenting Adversarial Mixup Samples

Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data

Towards In-Distribution Compatible Out-of-Distribution Detection.

Out-of-distribution Generalization via Partial Feature Decorrelation

Improving Out-of-Distribution Robustness of Classifiers via Generative Interpolation

Out-of-Distribution Generalization Via Style and Spuriousness Eliminating

Distribution Shift Inversion for Out-of-Distribution Prediction

DualAug: Exploiting Additional Heavy Augmentation with OOD Data Rejection

Towards Out-Of-Distribution Generalization: A Survey

InvariantOODG: Learning Invariant Features of Point Clouds for Out-of-Distribution Generalization

Make the Most of Your Data: Changing the Training Data Distribution to Improve In-distribution Generalization Performance

NormAUG: Normalization-guided Augmentation for Domain Generalization

Uncertainty Modeling for Out-of-Distribution Generalization