Abstract:Traditional emotion recognition technology often focuses on recognizing human biometrics such as facial expressions or body postures. However, psychological research shows that the context (contextual information) also plays an important role in perceiving the emotions of others. Existing research methods, that are based on contextual information, have heavily relied on the semantic features of images. They do not take into account the interrelationships between objects and fail to use external knowledge. Meanwhile, external knowledge is likely to be very helpful in perceiving emotion. In this paper, by incorporating external structured emotion commonsense knowledge, two methods are proposed for constructing emotion knowledge graphs based on the objective text of images, and a multi-modal emotion recognition model is designed. The model has three branches, one of which focuses on human biometrics, and another two branches employ emotion knowledge graphs to perceive emotion from contextual information. Before constructing the emotion knowledge graphs, we convert the visual content into the text information to obtain the prime and ample contextual information from the object, scene, and the relationship between the objects. This approach can reduce redundant and invalid information. After that, the structured emotion commonsense knowledge is integrated into the objective text in word sharing. A large-scale emotion knowledge graph based on all valid words (LEKG) and a small-scale emotion knowledge graph based on the document itself (TEKG) are constructed, respectively. We propose two fusion modules, one of which is attention-based, and the other is a deep reasoning module that incorporates interpersonal relation. We conduct extensive experiments on the benchmark dataset EMOTIC. The experimental results prove that our method is superior to the most advanced methods, and it has obvious advantages in global context-aware tasks.

Interpretable Emotion Analysis Based on Knowledge Graph and OCC Model

A emotion development agent model based on OCC model and operant conditioning

Incorporating Structured Emotion Commonsense Knowledge and Interpersonal Relation into Context-Aware Emotion Recognition

A domain-independent framework for modeling emotion

Semi-Automatic Complex Emotion Categorization and Ontology Construction from Chinese Knowledge

K-Act2Emo: Korean Commonsense Knowledge Graph for Indirect Emotional Expression

Human Emotion Knowledge Representation Emerges in Large Language Model and Supports Discrete Emotion Inference

Experiencer-Driven and Knowledge-Aware Graph Model for emotion-cause pair extraction

Emotion ontology construction from chinese knowledge

Appraisal Theories for Emotion Classification in Text

UNIFYING ALGORITHMIC AND THEORETICAL PERSPECTIVES: EMOTIONS IN ONLINE REVIEWS AND SALES

Dimensional Modeling of Emotions in Text with Appraisal Theories: Corpus Creation, Annotation Reliability, and Prediction

Emotion-based Analysis of Reviews using Knowledge Graph

EEG-data-oriented Knowledge Modeling and Emotion Recognition

Chinese Text-Oriented Interpersonal Emotional Relationship Mining Based on Event Knowledge Model

Enhancing emotion inference in conversations with commonsense knowledge

The Relationship Between Emotion Models and Artificial Intelligence

Image Emotion Analysis Based on Semantic Concepts

Emotion inference in conversations based on commonsense enhancement and graph structures

Building Ontology for Different Emotional Contexts and Multilingual Environment in Opinion Mining