Abstract:Identifying Out-of-distribution (OOD) data is becoming increasingly critical as the real-world applications of deep learning methods expand. Post-hoc methods modify softmax scores fine-tuned on outlier data or leverage intermediate feature layers to identify distinctive patterns between In-Distribution (ID) and OOD samples. Other methods focus on employing diverse OOD samples to learn discrepancies between ID and OOD. These techniques, however, are typically dependent on the quality of the outlier samples assumed. Density-based methods explicitly model class-conditioned distributions but this requires long training time or retraining the classifier. To tackle these issues, we introduce \textit{FlowCon}, a new density-based OOD detection technique. Our main innovation lies in efficiently combining the properties of normalizing flow with supervised contrastive learning, ensuring robust representation learning with tractable density estimation. Empirical evaluation shows the enhanced performance of our method across common vision datasets such as CIFAR-10 and CIFAR-100 pretrained on ResNet18 and WideResNet classifiers. We also perform quantitative analysis using likelihood plots and qualitative visualization using UMAP embeddings and demonstrate the robustness of the proposed method under various OOD contexts. Code will be open-sourced post decision.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to effectively detect out - of - distribution (OOD) data in deep - learning models. Specifically, the paper focuses on how to improve the model's ability to recognize OOD data by combining normalizing flows with supervised contrastive learning without relying on external OOD datasets. Traditional OOD detection methods either rely on external OOD datasets for tuning or perform poorly when dealing with near - OOD or far - OOD situations. The FlowCon method proposed in this paper aims to overcome these limitations and provide a more efficient and robust OOD detection technique.
### Main contributions
1. **Proposed a new density - based OOD detection technique**: Called FlowCon, a new loss function \( L_{\text{con}} \) is introduced, which enhances the compactness of the intra - class distribution and the separation of the inter - class distribution through contrastive learning.
2. **Performed well on multiple OOD detection metrics**: FlowCon was compared with existing methods on multiple metrics such as FPR95, AUROC, AUPR - Success, and AUPR - Error, showing its competitiveness or superiority under different OOD conditions.
3. **Provided detailed experimental results and visual analysis**: The ability of FlowCon in OOD detection and class preservation was demonstrated through histograms and UMAP embeddings, and its discriminability was shown through heat maps.
### Method overview
- **Normalizing Flows**: Used to transform high - dimensional features into a low - dimensional space with a well - defined probability distribution, so that the likelihood value of data points can be calculated.
- **Supervised Contrastive Learning**: By maximizing the similarity between samples of the same class and minimizing the similarity between samples of different classes, the model's representation learning ability is enhanced.
- **Joint loss function**: Combines the log - likelihood loss \( L_{\text{flow}} \) of normalizing flows and the loss \( L_{\text{con}} \) of contrastive learning, ensuring that the model can learn semantically meaningful representations and maintain traceable density estimates.
### Experimental setup
- **Datasets**: CIFAR - 10 and CIFAR - 100 were used as in - distribution (ID) datasets, and iSUN, LSUN - Crop, LSUN - Resize, SVHN, Textures, and Places365 were used as OOD datasets.
- **Pre - trained models**: ResNet18 and WideResNet were used as base classifiers.
- **Evaluation metrics**: AUROC, AUPR - Success, AUPR - Error, and FPR95.
### Experimental results
- **Far - OOD**: FlowCon performed well on both CIFAR - 10 and CIFAR - 100, especially on ResNet18, where the AUROC and AUPR - E metrics reached 97.19 and 85.65 respectively.
- **Mixed near - & far - OOD**: In the mixed OOD scenario of CIFAR - 10 and CIFAR - 100, FlowCon performed best on ResNet18, with AUPR - S reaching 98.74.
- **Near - OOD**: In the near - OOD scenario of CIFAR - 100 and CIFAR - 10, the FPR95 metric of FlowCon on ResNet18 reached 82.85, showing strong robustness.
Overall, through the FlowCon method, this paper provides a new way to effectively detect OOD data without relying on external OOD datasets, which has important theoretical and practical value.