Abstract:With the continuous development of deep learning (DL), the task of multimodal dialog emotion recognition (MDER) has recently received extensive research attention, which is also an essential branch of DL. The MDER aims to identify the emotional information contained in different modalities, e.g., text, video, and audio, and in different dialog scenes. However, the existing research has focused on modeling contextual semantic information and dialog relations between speakers while ignoring the impact of event relations on emotion. To tackle the above issues, we propose a novel dialog and event relation-aware graph convolutional neural network (DER-GCN) for multimodal emotion recognition method. It models dialog relations between speakers and captures latent event relations information. Specifically, we construct a weighted multirelationship graph to simultaneously capture the dependencies between speakers and event relations in a dialog. Moreover, we also introduce a self-supervised masked graph autoencoder (SMGAE) to improve the fusion representation ability of features and structures. Next, we design a new multiple information Transformer (MIT) to capture the correlation between different relations, which can provide a better fuse of the multivariate information between relations. Finally, we propose a loss optimization strategy based on contrastive learning to enhance the representation learning ability of minority class features. We conduct extensive experiments on the benchmark datasets, Interactive Emotional Dyadic Motion Capture (IEMOCAP) and Multimodal EmotionLines Dataset (MELD), which verify the effectiveness of the DER-GCN model. The results demonstrate that our model significantly improves both the average accuracy and the F1 value of emotion recognition. Our code is publicly available at https://github.com/yuntaoshou/DER-GCN.

Co-GAT: A Co-Interactive Graph Attention Network for Joint Dialog Act Recognition and Sentiment Classification

NGAT: Attention in Breadth and Depth Exploration for Semi-Supervised Graph Representation Learning

MLGAT: multi-layer graph attention networks for multimodal emotion recognition in conversations

Context- and Sentiment-Aware Networks for Emotion Recognition in Conversation

An Attentional Spatial Temporal Graph Convolutional Network with Co-Occurrence Feature Learning for Action Recognition

GoG: Relation-aware Graph-over-Graph Network for Visual Dialog.

Multi-task learning with graph attention networks for multi-domain task-oriented dialogue systems

GraphCFC: A Directed Graph Based Cross-Modal Feature Complementation Approach for Multimodal Conversational Emotion Recognition

CopGAT: Co-propagation Self-supervised Graph Attention Network

Dialogue Relation Extraction with Document-Level Heterogeneous Graph Attention Networks

GS-InGAT: An interaction graph attention network with global semantic for knowledge graph completion

GAR-Net: A Graph Attention Reasoning Network for conversation understanding

Emotion Recognition in Conversation Based on a Dynamic Complementary Graph Convolutional Network

Dual-channel and multi-granularity gated graph attention network for aspect-based sentiment analysis

GCAT-GTCU: Graph-Connected Attention Network and Gate Than Change Unit for Aspect-Level Sentiment Analysis

r-GAT: Relational Graph Attention Network for Multi-Relational Graphs

HAIN: Multi-label Classification with Hierarchical Attention-based Interaction Network for Multi-turn Dialogue Texts

Contextual Graph Attention Network for Aspect-Level Sentiment Classification

Co-attention networks based on aspect and context for aspect-level sentiment analysis

DER-GCN: Dialog and Event Relation-Aware Graph Convolutional Neural Network for Multimodal Dialog Emotion Recognition