Tracing Intricate Cues in Dialogue: Joint Graph Structure and Sentiment Dynamics for Multimodal Emotion Recognition

Jiang Li,Xiaoping Wang,Zhigang Zeng

2024-07-31

Abstract:Multimodal emotion recognition in conversation (MERC) has garnered substantial research attention recently. Existing MERC methods face several challenges: (1) they fail to fully harness direct inter-modal cues, possibly leading to less-than-thorough cross-modal modeling; (2) they concurrently extract information from the same and different modalities at each network layer, potentially triggering conflicts from the fusion of multi-source data; (3) they lack the agility required to detect dynamic sentimental changes, perhaps resulting in inaccurate classification of utterances with abrupt sentiment shifts. To address these issues, a novel approach named GraphSmile is proposed for tracking intricate emotional cues in multimodal dialogues. GraphSmile comprises two key components, i.e., GSF and SDP modules. GSF ingeniously leverages graph structures to alternately assimilate inter-modal and intra-modal emotional dependencies layer by layer, adequately capturing cross-modal cues while effectively circumventing fusion conflicts. SDP is an auxiliary task to explicitly delineate the sentiment dynamics between utterances, promoting the model's ability to distinguish sentimental discrepancies. Furthermore, GraphSmile is effortlessly applied to multimodal sentiment analysis in conversation (MSAC), forging a unified multimodal affective model capable of executing MERC and MSAC tasks. Empirical results on multiple benchmarks demonstrate that GraphSmile can handle complex emotional and sentimental patterns, significantly outperforming baseline models.

Computation and Language

What problem does this paper attempt to address?

The paper aims to address several key issues in Multimodal Emotion Recognition in Conversations (MERC):

1. **Insufficient Cross-Modal Cues**: Existing MERC methods fail to fully utilize direct cross-modal cues, resulting in incomplete cross-modal modeling.
2. **Conflict in Multi-Source Data Fusion**: Extracting both intra-modal and inter-modal information simultaneously at each network layer may lead to conflicts during information fusion.
3. **Inadequate Detection of Emotional Dynamics**: Current methods lack the ability to detect emotional dynamics, leading to inaccurate classification when handling sudden emotional shifts.

To tackle these issues, the authors propose a new method called GraphSmile, which addresses the problems through two core components:
- **Graph Structure Fusion (GSF) Module**: This module uses a graph structure to alternately fuse cross-modal relational cues and intra-modal contextual cues layer by layer, avoiding conflicts caused by simultaneous aggregation of different data sources.
- **Sentiment Dynamics Perception (SDP) Module**: As an auxiliary task, this module explicitly captures emotional dynamics in conversations, enhancing the model's ability to recognize emotional differences.

Additionally, GraphSmile extends to the Multimodal Sentiment Analysis in Conversations (MSAC) task and achieves joint optimization of both MERC and MSAC tasks, ultimately forming a unified multimodal sentiment model. Experimental results show that GraphSmile significantly outperforms baseline models on multiple benchmark datasets.