OOD Aware Supervised Contrastive Learning

Soroush Seifi,Daniel Olmeda Reino,Nikolay Chumerin,Rahaf Aljundi

2023-10-03

Abstract:Out-of-Distribution (OOD) detection is a crucial problem for the safe deployment of machine learning models identifying samples that fall outside of the training distribution, i.e. in-distribution data (ID). Most OOD works focus on the classification models trained with Cross Entropy (CE) and attempt to fix its inherent issues. In this work we leverage powerful representation learned with Supervised Contrastive (SupCon) training and propose a holistic approach to learn a classifier robust to OOD data. We extend SupCon loss with two additional contrast terms. The first term pushes auxiliary OOD representations away from ID representations without imposing any constraints on similarities among auxiliary data. The second term pushes OOD features far from the existing class prototypes, while pushing ID representations closer to their corresponding class prototype. When auxiliary OOD data is not available, we propose feature mixing techniques to efficiently generate pseudo-OOD features. Our solution is simple and efficient and acts as a natural extension of the closed-set supervised contrastive representation learning. We compare against different OOD detection methods on the common benchmarks and show state-of-the-art results.

Machine Learning,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the issue of detecting out-of-distribution (OOD) data in machine learning models, particularly in safety-critical applications such as autonomous driving systems. Specifically, the goal of the paper is to improve supervised contrastive learning methods to more effectively identify and handle OOD data.

The proposed method in the paper addresses the following key issues:
1. **Improving Supervised Contrastive Learning**: Utilizing the powerful representation capabilities of Supervised Contrastive Learning (SupCon) and proposing a new training objective based on it, namely OOD-Aware Supervised Contrastive Learning.
2. **Avoiding the Limitations of Cross-Entropy Loss**: Traditional models typically use cross-entropy loss to train classifiers, but this approach can lead to overly confident predictions on OOD data. Therefore, the proposed method does not rely on cross-entropy loss to learn classifier weights but instead learns class prototypes that reside in the same embedding space as the feature extractor.
3. **Enhancing OOD Data Handling**: Utilizing auxiliary or synthetic OOD data through two additional contrastive loss terms. The first loss term operates at the projection head level to minimize the similarity between in-distribution (ID) data and OOD data; the second loss term operates at the feature extractor level to push OOD data away from all learned class prototypes.
4. **Generating Pseudo-OOD Features**: When actual auxiliary OOD data is not available, a feature-mixing technique is proposed to generate pseudo-OOD features, thereby effectively regularizing the model.

In summary, the main contribution of the paper is the proposal of a new training framework that combines supervised contrastive learning with specific loss function designs to enhance the robustness and detection performance of models for OOD data.