Abstract:Facial expression recognition (FER) plays a crucial role in affective computing, enhancing human-computer interaction by enabling machines to understand and respond to human emotions. Despite advancements in deep learning, current FER systems often struggle with challenges such as occlusions, head pose variations, and motion blur in natural environments. These challenges highlight the need for more robust FER solutions. To address these issues, we propose the Attention-Enhanced Multi-Layer Transformer (AEMT) model, which integrates a dual-branch Convolutional Neural Network (CNN), an Attentional Selective Fusion (ASF) module, and a Multi-Layer Transformer Encoder (MTE) with transfer learning. The dual-branch CNN captures detailed texture and color information by processing RGB and Local Binary Pattern (LBP) features separately. The ASF module selectively enhances relevant features by applying global and local attention mechanisms to the extracted features. The MTE captures long-range dependencies and models the complex relationships between features, collectively improving feature representation and classification accuracy. Our model was evaluated on the RAF-DB and AffectNet datasets. Experimental results demonstrate that the AEMT model achieved an accuracy of 81.45% on RAF-DB and 71.23% on AffectNet, significantly outperforming existing state-of-the-art methods. These results indicate that our model effectively addresses the challenges of FER in natural environments, providing a more robust and accurate solution. The AEMT model significantly advances the field of FER by improving the robustness and accuracy of emotion recognition in complex real-world scenarios. This work not only enhances the capabilities of affective computing systems but also opens new avenues for future research in improving model efficiency and expanding multimodal data integration.

Modeling of Performance Creative Evaluation Driven by Multimodal Affective Data.

Modeling And Application Of Multimodal Affective User Interface With Multimedia

Multimodal Affective Computing Based on Weighted Linear Fusion

A Multimodal Emotion Perception Model Based on Context-Aware Decision-Level Fusion

Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment

Evaluation and analysis of visual perception using attention-enhanced computation in multimedia affective computing

Modelling the Influence of Personality and Culture on Affect and Enjoyment in Multimedia

Multitask Learning-Based Affective Prediction for Videos of Films and TV Scenes

Multimodal Prediction of Affective Dimensions Via Fusing Multiple Regression Techniques

EffMulti: Efficiently Modeling Complex Multimodal Interactions for Emotion Analysis

Affective Behaviour Analysis via Integrating Multi-Modal Knowledge

An Approach to Facial Expression Analysis with Multi-Model Interactions

Affective Video Content Analysis Based on Multimodal Data Fusion in Heterogeneous Networks.

Multimodal Fusion with Pre-Trained Model Features in Affective Behaviour Analysis In-the-wild

OPO-FCM: A Computational Affection Based OCC-PAD-OCEAN Federation Cognitive Modeling Approach

Performance Robots—Motion Control and Dynamic Emotion Regulation in Public Spaces

Facial Expression and Peripheral Physiology Fusion to Decode Individualized Affective Experience

Affective Experience Modeling Based on Interactive Synergetic Dependence in Big Data.

Multimodal Emotion Recognition Based on Feature Selection and Extreme Learning Machine in Video Clips.

Multi-modal Learning for Affective Content Analysis in Movies