Abstract:Out-of-Distribution (OOD) detection is crucial when deploying machine learning models in open-world applications. The core challenge in OOD detection is mitigating the model's overconfidence on OOD data. While recent methods using auxiliary outlier datasets or synthesizing outlier features have shown promising OOD detection performance, they are limited due to costly data collection or simplified assumptions. In this paper, we propose a novel OOD detection framework FodFoM that innovatively combines multiple foundation models to generate two types of challenging fake outlier images for classifier training.
The first type is based on BLIP-2's image captioning capability, CLIP's vision-language knowledge, and Stable Diffusion's image generation ability. Jointly utilizing these foundation models constructs fake outlier images which are semantically similar to but different from in-distribution (ID) images. For the second type, GroundingDINO's object detection ability is utilized to help construct pure background images by blurring foreground ID objects in ID images. The proposed framework can be flexibly combined with multiple existing OOD detection methods.
Extensive empirical evaluations show that image classifiers with the help of constructed fake images can more accurately differentiate real OOD images from ID ones.
New state-of-the-art OOD detection performance is achieved on multiple benchmarks. The code is available at \url{<a class="link-external link-https" href="https://github.com/Cverchen/ACMMM2024-FodFoM" rel="external noopener nofollow">this https URL</a>}.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is **how to improve the OOD (Out - of - Distribution) detection ability of machine learning models in open - world applications**. Specifically, the author points out that when deploying machine learning models, OOD detection is crucial because the model may be over - confident in OOD data, leading to misclassification. Although existing methods have made certain progress by using auxiliary abnormal data sets or synthesizing abnormal features, these methods have limitations such as high data collection costs or simplified assumptions.
To this end, the author proposes a new framework **FodFoM** (Fake Outlier Data by Foundation Models), which uses multiple foundation models to generate two types of fake abnormal images to train the classifier. These two types are:
1. **Fake abnormal images based on semantically similar but different ID images**: By combining the capabilities of BLIP - 2, CLIP and Stable Diffusion, generate fake abnormal images that are semantically similar to but different from ID images.
2. **Pure background images as fake abnormal images**: Utilize the object detection ability of GroundingDINO to blur the foreground objects in ID images and generate background images as fake abnormal images.
In this way, the FodFoM framework can help the classifier more accurately distinguish between real OOD images and ID images, thereby improving OOD detection performance. Experimental results show that this method has achieved the latest and best performance in multiple benchmark tests.
### Key Formulas
- **Supervised Contrastive Loss**:
\[
L_{SC}=-\frac{1}{N}\sum_{i = 1}^{N}\frac{1}{|P(i)|}\sum_{p\in P(i)}\log\frac{\exp(s(z_i,z_p)/\tau)}{\sum_{a\in A(i)}\exp(s(z_i,z_a)/\tau)}
\]
where \(N\) is the total number of training ID images and fake OOD images, \(A(i)\) represents the indices of all samples in the mini - batch containing the sample with index \(i\), \(P(i)\) is a subset of \(A(i)\) in which all corresponding samples have the same class label as the sample with index \(i\), \(z = g(f(x))\) is the projected visual embedding of the input image \(x\), \(s(z_i,z_p)\) represents the cosine similarity between the two embeddings \(z_i\) and \(z_p\), and \(\tau\) is the temperature scaling factor.
- **Joint Loss Function**:
\[
L = L_{CE}+\lambda L_{SC}
\]
where \(L_{CE}\) is the cross - entropy loss, \(L_{SC}\) is the supervised contrastive loss, and \(\lambda\) is a coefficient used to balance the two loss terms.
- **Energy - based OOD Detection Score**:
\[
E(x)=-\log\sum_{i = 1}^{C}\exp(h_i(f(x)))
\]
where \(h_i(f(x))\) represents the \(i\)-th logit value generated after the image \(x\) passes through the image encoder \(f\) and the classifier head \(h\). The energy value is defined as negative energy, that is, \(-E(x)\), and a larger score indicates that the image is more likely to come from a certain ID class.
Through these methods and formulas, the FodFoM framework significantly improves the performance of OOD detection.