Abstract:Most existing out-of-distribution (OOD) detection benchmarks classify samples with novel labels as the OOD data. However, some marginal OOD samples actually have close semantic contents to the in-distribution (ID) sample, which makes determining the OOD sample a Sorites Paradox. In this paper, we construct a benchmark named Incremental Shift OOD (IS-OOD) to address the issue, in which we divide the test samples into subsets with different semantic and covariate shift degrees relative to the ID dataset. The data division is achieved through a shift measuring method based on our proposed Language Aligned Image feature Decomposition (LAID). Moreover, we construct a Synthetic Incremental Shift (Syn-IS) dataset that contains high-quality generated images with more diverse covariate contents to complement the IS-OOD benchmark. We evaluate current OOD detection methods on our benchmark and find several important insights: (1) The performance of most OOD detection methods significantly improves as the semantic shift increases; (2) Some methods like GradNorm may have different OOD detection mechanisms as they rely less on semantic shifts to make decisions; (3) Excessive covariate shifts in the image are also likely to be considered as OOD for some methods. Our code and data are released in <a class="link-external link-https" href="https://github.com/qqwsad5/IS-OOD" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the defects existing in the existing Out - of - Distribution (OOD) detection benchmarks. That is, these benchmarks usually rely on semantic labels to distinguish OOD samples, while ignoring the similarity in semantic content between some marginal OOD samples and In - Distribution (ID) samples. This problem is similar to the "Sorites Paradox", that is, it is difficult to precisely define how much difference a sample needs to have from ID data to be regarded as an OOD sample. Specifically, the author points out: 1. **Limitations of existing benchmarks**: Most of the existing OOD detection benchmarks classify samples with novel labels as OOD data, but some marginal OOD samples are actually very close to ID samples in semantic content, which makes it difficult to determine OOD samples. 2. **The problem of inaccurate labels**: The semantic labels of images are sometimes inaccurate and may miss background objects or common - sense relationships, resulting in label noise (for example, the labels of "African forest elephant" and "African elephant" are similar, or the labels of "corn" and "food" overlap). 3. **Lack of a systematic method to measure differences**: Most of the current research focuses on whether a sample is OOD, rather than measuring the degree of difference between the sample and ID data. To solve these problems, the author proposes a new benchmark - **Incremental Shift OOD (IS - OOD)**, and introduces the following methods: - **Constructing the IS - OOD benchmark**: By dividing test samples into subsets with different degrees of semantic and covariate shift, it avoids directly relying on semantic labels to judge whether a sample is OOD. - **Proposing the LAID method**: Based on the alignment characteristics of the CLIP model, the Language Aligned Image feature Decomposition (LAID) method is proposed to decompose image features, so as to measure semantic and covariate shift respectively. - **Generating the Syn - IS dataset**: In order to enhance the diversity of covariate components, a Synthetic Incremental Shift (Syn - IS) dataset containing high - quality generated images is constructed to supplement the IS - OOD benchmark. Through these improvements, the author hopes to provide a more systematic and comprehensive way to evaluate the performance of OOD detection methods and reveals some important findings, such as: 1. The performance of most OOD detection methods significantly improves as the semantic shift increases. 2. Some methods (such as GradNorm) may have different OOD detection mechanisms because they rely less on semantic shift to make decisions. 3. Excessive covariate shift in an image may also be regarded as OOD by some methods. These findings are helpful for better understanding the working mechanism of OOD detection methods and provide valuable references for future research.

Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites Paradox

ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms

Dissecting Out-of-Distribution Detection and Open-Set Recognition: A Critical Analysis of Methods and Benchmarks

Recent Advances in OOD Detection: Problems and Approaches

OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection

The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection

Rethinking Out-of-Distribution Detection on Imbalanced Data Distribution

Out-of-distribution Detection with Implicit Outlier Transformation

Out-of-Distribution Learning with Human Feedback

Rethinking Out-of-Distribution Detection From a Human-Centric Perspective

Re-Benchmarking Out-of-Distribution Detection in Deep Neural Networks

OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization

Out-of-Distribution Detection with Virtual Outlier Smoothing

Out-of-Distribution Detection with Overlap Index

Out-of-Distribution Detection with Hilbert-Schmidt Independence Optimization

Going Beyond Conventional OOD Detection

A Survey on Evaluation of Out-of-Distribution Generalization

A Survey on Out-of-Distribution Detection in NLP

OAL: Enhancing OOD Detection Using Latent Diffusion

Out-Of-Distribution Detection with Diversification (Provably)