Abstract:The emotion of humans is an important indicator or reflection of their mental states, e.g., satisfaction or stress, and recognizing or detecting emotion from different media is essential to perform sequence analysis or for certain applications, e.g., mental health assessments, job stress level estimation, and tourist satisfaction assessments. Emotion recognition based on computer vision techniques, as an important method of detecting emotion from visual media (e.g., images or videos) of human behaviors with the use of plentiful emotional cues, has been extensively investigated because of its significant applications. However, most existing models neglect inter-feature interaction and use simple concatenation for feature fusion, failing to capture the crucial complementary gains between face and context information in video clips, which is significant in addressing the problems of emotion confusion and emotion misunderstanding. Accordingly, in this paper, to fully exploit the complementary information between face and context features, we present a novel cross-attention and hybrid feature weighting network to achieve accurate emotion recognition from large-scale video clips, and the proposed model consists of a dual-branch encoding (DBE) network, a hierarchical-attention encoding (HAE) network, and a deep fusion (DF) block. Specifically, the face and context encoding blocks in the DBE network generate the respective shallow features. After this, the HAE network uses the cross-attention (CA) block to investigate and capture the complementarity between facial expression features and their contexts via a cross-channel attention operation. The element recalibration (ER) block is introduced to revise the feature map of each channel by embedding global information. Moreover, the adaptive-attention (AA) block in the HAE network is developed to infer the optimal feature fusion weights and obtain the adaptive emotion features via a hybrid feature weighting operation. Finally, the DF block integrates these adaptive emotion features to predict an individual emotional state. Extensive experimental results of the CAER-S dataset demonstrate the effectiveness of our method, exhibiting its potential in the analysis of tourist reviews with video clips, estimation of job stress levels with visual emotional evidence, or assessments of mental healthiness with visual media.

Video-EEG Based Collaborative Emotion Recognition Using LSTM and Information-Attention

JDAT: Joint-Dimension-Aware Transformer with Strong Flexibility for EEG Emotion Recognition

EEG Emotion Recognition Based on Self-attention Dynamic Graph Neural Networks

Multi-view Domain-Adaptive Representation Learning for EEG-based Emotion Recognition

An Efficient LSTM Network for Emotion Recognition from Multichannel EEG Signals

Exploiting EEG signals and audiovisual feature fusion for video emotion recognition

Channel attention convolutional aggregation network based on video-level features for EEG emotion recognition

Using electroencephalogram for emotion recognition based on filter-bank long short-term memory networks

Emotion Recognition from Large-Scale Video Clips with Cross-Attention and Hybrid Feature Weighting Neural Networks

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Expression EEG Multimodal Emotion Recognition Method Based on the Bidirectional LSTM and Attention Mechanism

EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM

Long Short Term Memory Recurrent Neural Network Based Encoding Method for Emotion Recognition in Video.

An attention-based hybrid deep learning model for EEG emotion recognition

Temporal aware Mixed Attention-based Convolution and Transformer Network for cross-subject EEG emotion recognition

A Model for EEG-Based Emotion Recognition: CNN-Bi-LSTM with Attention Mechanism

A Novel DE-CNN-BiLSTM Multi-Fusion Model for EEG Emotion Recognition

CNN and LSTM based ensemble learning for human emotion recognition using EEG recordings

Valence-Arousal Model based Emotion Recognition using EEG, peripheral physiological signals and Facial Expression

S-LSTM-ATT: a hybrid deep learning approach with optimized features for emotion recognition in electroencephalogram

Expression-Guided EEG Representation Learning for Emotion Recognition