Abstract:Traditional emotion recognition technology often focuses on recognizing human biometrics such as facial expressions or body postures. However, psychological research shows that the context (contextual information) also plays an important role in perceiving the emotions of others. Existing research methods, that are based on contextual information, have heavily relied on the semantic features of images. They do not take into account the interrelationships between objects and fail to use external knowledge. Meanwhile, external knowledge is likely to be very helpful in perceiving emotion. In this paper, by incorporating external structured emotion commonsense knowledge, two methods are proposed for constructing emotion knowledge graphs based on the objective text of images, and a multi-modal emotion recognition model is designed. The model has three branches, one of which focuses on human biometrics, and another two branches employ emotion knowledge graphs to perceive emotion from contextual information. Before constructing the emotion knowledge graphs, we convert the visual content into the text information to obtain the prime and ample contextual information from the object, scene, and the relationship between the objects. This approach can reduce redundant and invalid information. After that, the structured emotion commonsense knowledge is integrated into the objective text in word sharing. A large-scale emotion knowledge graph based on all valid words (LEKG) and a small-scale emotion knowledge graph based on the document itself (TEKG) are constructed, respectively. We propose two fusion modules, one of which is attention-based, and the other is a deep reasoning module that incorporates interpersonal relation. We conduct extensive experiments on the benchmark dataset EMOTIC. The experimental results prove that our method is superior to the most advanced methods, and it has obvious advantages in global context-aware tasks.

Exploiting Emotional Concepts for Image Emotion Recognition

Image Emotion Analysis Based on Semantic Concepts

Image Emotional Semantic Retrieval Based on ELM

Emotion Recognition via Environmental Context and Human Body

Image Emotion Computing

Incorporating Structured Emotion Commonsense Knowledge and Interpersonal Relation into Context-Aware Emotion Recognition

Concept-guided multi-level attention network for image emotion recognition

Visual Emotion Analysis Via Affective Semantic Concept Discovery

Predicting Personalized Emotion Perceptions of Social Images

A Multi-Stage Visual Perception Approach for Image Emotion Analysis

Recognizing Image Semantic Information Through Multi-Feature Fusion and SSAE-Based Deep Network

Learning Multi-level Representations for Image Emotion Recognition in the Deep Convolutional Network

Predicting Personalized Image Emotion Perceptions in Social Networks

Feature Discrimination and Diversification for Image Emotion Recognition

Affective Computing of Image Emotion Perceptions

Exploiting Visual Context and Multi-grained Semantics for Social Text Emotion Recognition

Exploration meets exploitation: Multitask learning for emotion recognition based on discrete and dimensional models

A Method of the Image Emotion Notation Based Fuzzy Reasoning

How Do Your Friends on Social Media Disclose Your Emotions?

Dependency Exploitation: A Unified CNN-RNN Approach for Visual Emotion Recognition

Image Emotion Multi-Label Classification Based on Multi-Graph Learning