Abstract:Facial expression recognition (FER) plays a crucial role in affective computing, enhancing human-computer interaction by enabling machines to understand and respond to human emotions. Despite advancements in deep learning, current FER systems often struggle with challenges such as occlusions, head pose variations, and motion blur in natural environments. These challenges highlight the need for more robust FER solutions. To address these issues, we propose the Attention-Enhanced Multi-Layer Transformer (AEMT) model, which integrates a dual-branch Convolutional Neural Network (CNN), an Attentional Selective Fusion (ASF) module, and a Multi-Layer Transformer Encoder (MTE) with transfer learning. The dual-branch CNN captures detailed texture and color information by processing RGB and Local Binary Pattern (LBP) features separately. The ASF module selectively enhances relevant features by applying global and local attention mechanisms to the extracted features. The MTE captures long-range dependencies and models the complex relationships between features, collectively improving feature representation and classification accuracy. Our model was evaluated on the RAF-DB and AffectNet datasets. Experimental results demonstrate that the AEMT model achieved an accuracy of 81.45% on RAF-DB and 71.23% on AffectNet, significantly outperforming existing state-of-the-art methods. These results indicate that our model effectively addresses the challenges of FER in natural environments, providing a more robust and accurate solution. The AEMT model significantly advances the field of FER by improving the robustness and accuracy of emotion recognition in complex real-world scenarios. This work not only enhances the capabilities of affective computing systems but also opens new avenues for future research in improving model efficiency and expanding multimodal data integration.

Attention on Emotions: A Vision Transformer Approach to Advancing Facial Expression Recognition

Comparative Analysis of Vision Transformer Models for Facial Emotion Recognition Using Augmented Balanced Datasets

Facial Expression Recognition Based on Multi-Scale Convolutional Vision Transformer

Facial Expression Recognition With Visual Transformers and Attentional Selective Fusion

An enhanced speech emotion recognition using vision transformer

Facial Expression Recognition using Squeeze and Excitation-powered Swin Transformers

MViT: Mask Vision Transformer for Facial Expression Recognition in the Wild

MFEViT: A Robust Lightweight Transformer-based Network for Multimodal 2D+3D Facial Expression Recognition

Personalized Speech Emotion Recognition in Human-Robot Interaction using Vision Transformers

Emotion Recognition Using Transformers with Masked Learning

Transformer-Based Multimodal Emotional Perception for Dynamic Facial Expression Recognition in the Wild

Facial Expression Recognition Based on Fine-Tuned Channel–Spatial Attention Transformer

VaBTFER: An Effective Variant Binary Transformer for Facial Expression Recognition

Emotion-Aware Transformer Encoder for Empathetic Dialogue Generation

ViTCN: Hybrid Vision Transformer with Temporal Convolution for Multi-Emotion Recognition

A novel facial emotion recognition model using segmentation VGG-19 architecture

Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network

Emotion Separation and Recognition from a Facial Expression by Generating the Poker Face with Vision Transformers

Enhanced Hybrid Vision Transformer with Multi-Scale Feature Integration and Patch Dropping for Facial Expression Recognition

Evaluation and analysis of visual perception using attention-enhanced computation in multimedia affective computing