Representation Norm Amplification for Out-of-Distribution Detection in Long-Tail Learning

Dong Geun Shin,Hye Won Chung

2024-08-20

Abstract:Detecting out-of-distribution (OOD) samples is a critical task for reliable machine learning. However, it becomes particularly challenging when the models are trained on long-tailed datasets, as the models often struggle to distinguish tail-class in-distribution samples from OOD samples. We examine the main challenges in this problem by identifying the trade-offs between OOD detection and in-distribution (ID) classification, faced by existing methods. We then introduce our method, called \textit{Representation Norm Amplification} (RNA), which solves this challenge by decoupling the two problems. The main idea is to use the norm of the representation as a new dimension for OOD detection, and to develop a training method that generates a noticeable discrepancy in the representation norm between ID and OOD data, while not perturbing the feature learning for ID classification. Our experiments show that RNA achieves superior performance in both OOD detection and classification compared to the state-of-the-art methods, by 1.70\% and 9.46\% in FPR95 and 2.43\% and 6.87\% in classification accuracy on CIFAR10-LT and ImageNet-LT, respectively. The code for this work is available at <a class="link-external link-https" href="https://github.com/dgshin21/RNA" rel="external noopener nofollow">this https URL</a>.

Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the reliable out - of - distribution (OOD) abnormal sample detection on long - tailed distribution datasets. Specifically, when machine - learning models are trained on long - tailed distribution datasets, these models are often difficult to distinguish normal samples (in - distribution, ID) from abnormal samples (OOD) in the tail categories. This is mainly because in long - tailed distribution datasets, the number of samples in some categories is very small, resulting in insufficient learning of these category samples by the model, thus affecting the model's judgment ability on these samples. Therefore, the main objective of the paper is to propose a method that can improve the model's detection ability of abnormal samples without affecting the model's classification performance of normal samples in long - tailed distribution datasets.

To meet this challenge, the paper introduces a new method, called Representation Norm Amplification (RNA), to solve the above problems by decoupling the classification task and the abnormal detection task. The core idea of the RNA method is to use the norm of the representation vector as a new dimension for abnormal detection and develop a training method to make a significant difference between the representation norms of normal samples and abnormal samples while not interfering with the learning of normal sample classification features. Experimental results show that the RNA method is superior to the existing state - of - the - art methods in both abnormal detection and classification performance.