Open-Set Facial Expression Recognition

Yuhang Zhang,Yue Yao,Xuannan Liu,Lixiong Qin,Wenjing Wang,Weihong Deng

2024-01-23

Abstract:Facial expression recognition (FER) models are typically trained on datasets with a fixed number of seven basic classes. However, recent research works point out that there are far more expressions than the basic ones. Thus, when these models are deployed in the real world, they may encounter unknown classes, such as compound expressions that cannot be classified into existing basic classes. To address this issue, we propose the open-set FER task for the first time. Though there are many existing open-set recognition methods, we argue that they do not work well for open-set FER because FER data are all human faces with very small inter-class distances, which makes the open-set samples very similar to close-set samples. In this paper, we are the first to transform the disadvantage of small inter-class distance into an advantage by proposing a new way for open-set FER. Specifically, we find that small inter-class distance allows for sparsely distributed pseudo labels of open-set samples, which can be viewed as symmetric noisy labels. Based on this novel observation, we convert the open-set FER to a noisy label detection problem. We further propose a novel method that incorporates attention map consistency and cycle training to detect the open-set samples. Extensive experiments on various FER datasets demonstrate that our method clearly outperforms state-of-the-art open-set recognition methods by large margins. Code is available at

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the problem of achieving open-set recognition in facial expression recognition (FER). Traditional FER models are typically trained on datasets containing 7 basic categories, but in real-world applications, humans may exhibit more expressions beyond these basic categories. Therefore, when these models are deployed in the real world, they may encounter unknown categories, such as compound expressions that cannot be classified into existing basic categories. To solve this problem, the paper proposes the open-set FER task for the first time, aiming to enable the model to not only maintain high accuracy in recognizing known categories but also identify samples belonging to unknown categories.

However, due to the very small inter-class distance of facial images in FER datasets, open-set samples are very similar to closed-set samples, which causes existing open-set recognition methods to perform poorly in this scenario. To address this issue, the paper proposes a new approach that transforms the open-set FER into a noise label detection problem and combines attention map consistency and cycle training to effectively separate open-set and closed-set samples. Experimental results show that this method significantly outperforms the current state-of-the-art open-set recognition methods on multiple open-set FER datasets.