Abstract:Facial expression recognition (FER) plays a crucial role in affective computing, enhancing human-computer interaction by enabling machines to understand and respond to human emotions. Despite advancements in deep learning, current FER systems often struggle with challenges such as occlusions, head pose variations, and motion blur in natural environments. These challenges highlight the need for more robust FER solutions. To address these issues, we propose the Attention-Enhanced Multi-Layer Transformer (AEMT) model, which integrates a dual-branch Convolutional Neural Network (CNN), an Attentional Selective Fusion (ASF) module, and a Multi-Layer Transformer Encoder (MTE) with transfer learning. The dual-branch CNN captures detailed texture and color information by processing RGB and Local Binary Pattern (LBP) features separately. The ASF module selectively enhances relevant features by applying global and local attention mechanisms to the extracted features. The MTE captures long-range dependencies and models the complex relationships between features, collectively improving feature representation and classification accuracy. Our model was evaluated on the RAF-DB and AffectNet datasets. Experimental results demonstrate that the AEMT model achieved an accuracy of 81.45% on RAF-DB and 71.23% on AffectNet, significantly outperforming existing state-of-the-art methods. These results indicate that our model effectively addresses the challenges of FER in natural environments, providing a more robust and accurate solution. The AEMT model significantly advances the field of FER by improving the robustness and accuracy of emotion recognition in complex real-world scenarios. This work not only enhances the capabilities of affective computing systems but also opens new avenues for future research in improving model efficiency and expanding multimodal data integration.

EERCA-ViT: Enhanced Effective Region and Context-Aware Vision Transformers for Image Sentiment Analysis

Visual sentiment analysis with semantic correlation enhancement

Visual Sentiment Prediction Based on Automatic Discovery of Affective Regions

Multi-scale Features Enhanced Sentiment Region Discovery for Visual Sentiment Analysis

Image Sentiment Classification Via Multi-Level Sentiment Region Correlation Analysis.

Emotion Recognition via Environmental Context and Human Body

Visual contextual relationship augmented transformer for image captioning

Facial Expression Recognition with Visual Transformers and Attentional Selective Fusion

Evaluation and analysis of visual perception using attention-enhanced computation in multimedia affective computing

Transformer-Based Multimodal Emotional Perception for Dynamic Facial Expression Recognition in the Wild

Emotion Transformer Fusion - Complementary Representation Properties of EEG and Eye Movements on Recognizing Anger and Surprise.

Discovering Affective Regions in Deep Convolutional Neural Networks for Visual Sentiment Prediction.

Attention on Emotions: A Vision Transformer Approach to Advancing Facial Expression Recognition

Stimuli-Aware Visual Emotion Analysis

A Multi-Attentive Pyramidal Model for Visual Sentiment Analysis

Sentimental Visual Captioning Using Multimodal Transformer

Context Transformer and Adaptive Method with Visual Transformer for Robust Facial Expression Recognition

Multi-depth Fusion Transformer and Batch Piecewise Loss for Visual Sentiment Analysis.

An enhanced speech emotion recognition using vision transformer

A Self-Attention Based Method for Facial Expression Recognition