Abstract:Facial expression data is characterized by a significant imbalance, with most collected data showing happy or neutral expressions and fewer instances of fear or disgust. This imbalance poses challenges to facial expression recognition (FER) models, hindering their ability to fully understand various human emotional states. Existing FER methods typically report overall accuracy on highly imbalanced test sets but exhibit low performance in terms of the mean accuracy across all expression classes. In this paper, our aim is to address the imbalanced FER problem. Existing methods primarily focus on learning knowledge of minor classes solely from minor-class samples. However, we propose a novel approach to extract extra knowledge related to the minor classes from both major and minor class samples. Our motivation stems from the belief that FER resembles a distribution learning task, wherein a sample may contain information about multiple classes. For instance, a sample from the major class surprise might also contain useful features of the minor class fear. Inspired by that, we propose a novel method that leverages re-balanced attention maps to regularize the model, enabling it to extract transformation invariant information about the minor classes from all training samples. Additionally, we introduce re-balanced smooth labels to regulate the cross-entropy loss, guiding the model to pay more attention to the minor classes by utilizing the extra information regarding the label distribution of the imbalanced training data. Extensive experiments on different datasets and backbones show that the two proposed modules work together to regularize the model and achieve state-of-the-art performance under the imbalanced FER task. Code is available at <a class="link-external link-https" href="https://github.com/zyh-uaiaaaa" rel="external noopener nofollow">this https URL</a>.

RASN: Using Attention and Sharing Affinity Features to Address Sample Imbalance in Facial Expression Recognition

SDNET: Lightweight Facial Expression Recognition For Sample Disequilibrium.

Multi-Head Attention Affinity Diversity Sharing Network for Facial Expression Recognition

Efficient Facial Expression Recognition with Representation Reinforcement Network and Transfer Self-Training for Human–Machine Interaction

BFFN: A Novel Balanced Feature Fusion Network for Fair Facial Expression Recognition

A New Joint Training Method for Facial Expression Recognition with Inconsistently Annotated and Imbalanced Data

Uncertain and Biased Facial Expression Recognition Based on Depthwise Separable Convolutional Neural Network with Embedded Attention Mechanism

Combating Uncertainty and Class Imbalance in Facial Expression Recognition

An Improved SimAM Based CNN for Facial Expression Recognition

Balancing the Scales: Enhancing Fairness in Facial Expression Recognition with Latent Alignment

Expression Recognition Based on Attention-Split Convolutional Residual Network

Adaptively Learning Facial Expression Representation via C-F Labels and Distillation

Accurate and Reliable Facial Expression Recognition Using Advanced Softmax Loss with Fixed Weights

Hybrid Attention-Aware Learning Network for Facial Expression Recognition in the Wild

Blindfold Attention: Novel Mask Strategy for Facial Expression Recognition

Leave No Stone Unturned: Mine Extra Knowledge for Imbalanced Facial Expression Recognition

Mixing Global and Local Features for Long-Tailed Expression Recognition

A Multi-Head Convolution Network with Attention Consistency for Facial Expression Recognition

Combination of Valence-Sensitive Loss with Restrictive Center Loss for Facial Expression Recognition

K-Face Net: A Two-Stage Framework for Balanced Feature Space in Facial Expression Recognition

Investigating Bias and Fairness in Facial Expression Recognition