Abstract:Multi-modal Emotion Recognition in Conversation (MERC) has received considerable attention in various fields, e.g., human-computer interaction and recommendation systems. Most existing works perform feature disentanglement and fusion to extract emotional contextual information from multi-modal features and emotion classification. After revisiting the characteristic of MERC, we argue that long-range contextual semantic information should be extracted in the feature disentanglement stage and the inter-modal semantic information consistency should be maximized in the feature fusion stage. Inspired by recent State Space Models (SSMs), Mamba can efficiently model long-distance dependencies. Therefore, in this work, we fully consider the above insights to further improve the performance of MERC. Specifically, on the one hand, in the feature disentanglement stage, we propose a Broad Mamba, which does not rely on a self-attention mechanism for sequence modeling, but uses state space models to compress emotional representation, and utilizes broad learning systems to explore the potential data distribution in broad space. Different from previous SSMs, we design a bidirectional SSM convolution to extract global context information. On the other hand, we design a multi-modal fusion strategy based on probability guidance to maximize the consistency of information between modalities. Experimental results show that the proposed method can overcome the computational and memory limitations of Transformer when modeling long-distance contexts, and has great potential to become a next-generation general architecture in MERC.

A Bi-directional Multi-hop Inference Model for Joint Dialog Sentiment Classification and Act Recognition

Hierarchical and Bidirectional Joint Multi-Task Classifiers for Natural Language Understanding

Sentiment Analysis Using Deep Robust Complementary Fusion of Multi-Features and Multi-Modalities.

DARER: Dual-task Temporal Relational Recurrent Reasoning Network for Joint Dialog Sentiment Classification and Act Recognition

DCR-Net: A Deep Co-Interactive Relation Network for Joint Dialog Act Recognition and Sentiment Classification.

A study of sentiment classification methods with dual model decision fusion

Combining Heterogeneous Deep Neural Networks with Conditional Random Fields for Chinese Dialogue Act Recognition

Multi-level Multi-task Representation Learning with Adaptive Fusion for Multimodal Sentiment Analysis

DMRM: A Dual-Channel Multi-Hop Reasoning Model for Visual Dialog

Self-adaptive Context and Modal-interaction Modeling For Multimodal Emotion Recognition

Early Joint Learning of Emotion Information Makes MultiModal Model Understand You Better

AIMDiT: Modality Augmentation and Interaction via Multimodal Dimension Transformation for Emotion Recognition in Conversations

Multimodal interaction enhanced representation learning for video emotion recognition

Sentiment Boosting Model for Emotion Recognition in Conversation Text

Tracing Intricate Cues in Dialogue: Joint Graph Structure and Sentiment Dynamics for Multimodal Emotion Recognition

Dual Causes Generation Assisted Model for Multimodal Aspect-Based Sentiment Classification

Exploring Multimodal Sentiment Analysis via CBAM Attention and Double-layer BiLSTM Architecture

Cross-Modal Sentiment Sensing with Visual-Augmented Representation and Diverse Decision Fusion

Revisiting Multi-modal Emotion Learning with Broad State Space Models and Probability-guidance Fusion

SMIN: Semi-supervised Multi-modal Interaction Network for Conversational Emotion Recognition

A Deep Spatiotemporal Interaction Network for Multimodal Sentimental Analysis and Emotion Recognition