Abstract:Traditional machine learning follows a close-set assumption that the training and test set share the same label space. While in many practical scenarios, it is inevitable that some test samples belong to unknown classes (open-set). To fix this issue, Open-Set Recognition (OSR), whose goal is to make correct predictions on both close-set samples and open-set samples, has attracted rising attention. In this direction, the vast majority of literature focuses on the pattern of open-set samples. However, how to evaluate model performance in this challenging task is still unsolved. In this paper, a systematic analysis reveals that most existing metrics are essentially inconsistent with the aforementioned goal of OSR: (1) For metrics extended from close-set classification, such as Open-set F-score, Youden's index, and Normalized Accuracy, a poor open-set prediction can escape from a low performance score with a superior close-set prediction. (2) Novelty detection AUC, which measures the ranking performance between close-set and open-set samples, ignores the close-set performance. To fix these issues, we propose a novel metric named OpenAUC. Compared with existing metrics, OpenAUC enjoys a concise pairwise formulation that evaluates open-set performance and close-set performance in a coupling manner. Further analysis shows that OpenAUC is free from the aforementioned inconsistency properties. Finally, an end-to-end learning method is proposed to minimize the OpenAUC risk, and the experimental results on popular benchmark datasets speak to its effectiveness. Project Page: <a class="link-external link-https" href="https://github.com/wang22ti/OpenAUC" rel="external noopener nofollow">this https URL</a>.

New multiple-classifier appraisal method based on AUC

AdaBoost-based Multi-attribute Classification Technology and Its Application

Multiclass ROC

A New Computer-Aided Method for Diagnosis of Breast Cancer Based on Decision Tree

Comparing multi-class classifier performance by multi-class ROC analysis: A nonparametric approach

MFA-DMFS:new Multi-Classifier Fusion Method and Its Application

A Modified AUC for Training Convolutional Neural Networks: Taking Confidence Into Account

An Innovative Method to Classify Remote Sensing Images Using Artificial Bee Colony Algorithm

A Novel Multi-class Classification Architecture Combining Population-based Sampling and Multi-expert Classifier for Imbalanced Data.

Multiple attribute decision making based on score function of q-connection numbers, q-CNPWG aggregation operator of q-connection numbers, and set pair analysis theory in the environments of q-rung orthopair fuzzy numbers

Two-Stage Multi-Class Adaboost for Facial Expression Recognition

A Closer Look at AUROC and AUPRC under Class Imbalance

An efficient variance estimator of AUC and its applications to binary classification

AUCμ: A Performance Metric for Multi-Class Machine Learning Models

OpenAUC: Towards AUC-Oriented Open-Set Recognition

A Modified Random Forest Based on Kappa Measure and Binary Artificial Bee Colony Algorithm

Dual Coordinate Descent Method for Solving AUC Optimization Problem

A Non-Parametric Method for the Comparison of Partial Areas under ROC Curves and Its Application to Large Health Care Data Sets.

Empirical Comparison of Area under ROC curve (AUC) and Mathew Correlation Coefficient (MCC) for Evaluating Machine Learning Algorithms on Imbalanced Datasets for Binary Classification

Multiple Criteria Inventory Classification Based on Principal Components Analysis and Neural Network

The MCC-F1 curve: a performance evaluation technique for binary classification