Abstract:The emotion of humans is an important indicator or reflection of their mental states, e.g., satisfaction or stress, and recognizing or detecting emotion from different media is essential to perform sequence analysis or for certain applications, e.g., mental health assessments, job stress level estimation, and tourist satisfaction assessments. Emotion recognition based on computer vision techniques, as an important method of detecting emotion from visual media (e.g., images or videos) of human behaviors with the use of plentiful emotional cues, has been extensively investigated because of its significant applications. However, most existing models neglect inter-feature interaction and use simple concatenation for feature fusion, failing to capture the crucial complementary gains between face and context information in video clips, which is significant in addressing the problems of emotion confusion and emotion misunderstanding. Accordingly, in this paper, to fully exploit the complementary information between face and context features, we present a novel cross-attention and hybrid feature weighting network to achieve accurate emotion recognition from large-scale video clips, and the proposed model consists of a dual-branch encoding (DBE) network, a hierarchical-attention encoding (HAE) network, and a deep fusion (DF) block. Specifically, the face and context encoding blocks in the DBE network generate the respective shallow features. After this, the HAE network uses the cross-attention (CA) block to investigate and capture the complementarity between facial expression features and their contexts via a cross-channel attention operation. The element recalibration (ER) block is introduced to revise the feature map of each channel by embedding global information. Moreover, the adaptive-attention (AA) block in the HAE network is developed to infer the optimal feature fusion weights and obtain the adaptive emotion features via a hybrid feature weighting operation. Finally, the DF block integrates these adaptive emotion features to predict an individual emotional state. Extensive experimental results of the CAER-S dataset demonstrate the effectiveness of our method, exhibiting its potential in the analysis of tourist reviews with video clips, estimation of job stress levels with visual emotional evidence, or assessments of mental healthiness with visual media.

Boosting Image Emotion Analysis Via Coupling Dependent Attentive Model.

Exploring Affective Image Representation with Visual Attention and Aesthetic Fusion

Emotion Recognition via Environmental Context and Human Body

Deep Emotional Arousal Network for Multimodal Sentiment Analysis and Emotion Recognition

Affective Image Recognition with Multi-Attribute Knowledge in Deep Neural Networks

Multi-Output Learning Based on Multimodal GCN and Co-Attention for Image Aesthetics and Emotion Analysis

Recognizing Image Semantic Information Through Multi-Feature Fusion and SSAE-Based Deep Network

Image Captioning with Emotional Information Via Multiple Model

Emotion Recognition from Large-Scale Video Clips with Cross-Attention and Hybrid Feature Weighting Neural Networks

Learning multi-level representations for affective image recognition

Image Aesthetics Assessment With Emotion-Aware Multibranch Network

Image Aesthetic Assessment Based on Emotion-Assisted Multi-Task Learning Network

Image Emotion Caption Based on Visual Attention Mechanisms

Image Aesthetics Assessment Based on Hypernetwork of Emotion Fusion

Image Emotion Analysis Based on Semantic Concepts

SCEP—A New Image Dimensional Emotion Recognition Model Based on Spatial and Channel-Wise Attention Mechanisms

ARMNet: A Network for Image Dimensional Emotion Prediction Based on Affective Region Extraction and Multi-Channel Fusion

Learning Multi-level Representations for Image Emotion Recognition in the Deep Convolutional Network

A Multi-feature Fusion and SSAE-Based Deep Network for Image Semantic Recognition

Fine-grained Dual-space Context Analysis Network for Multimodal Emotion Recognition

Towards Unified Aesthetics and Emotion Prediction in Images.