Abstract:Multimodal emotion recognition in conversation (MER) aims to accurately identify emotions in conversational utterances by integrating multimodal information. Previous methods usually treat multimodal information as equal quality and employ symmetric architectures to conduct multimodal fusion. However, in reality, the quality of different modalities usually varies considerably, and utilizing a symmetric architecture is difficult to accurately recognize conversational emotions when dealing with uneven modal information. Furthermore, fusing multi-modality information in a single granularity may fail to adequately integrate modal information, exacerbating the inaccuracy in emotion recognition. In this paper, we propose a novel Cross-Modality Augmented Transformer with Hierarchical Variational Distillation, called CMATH, which consists of two major components, i.e., Multimodal Interaction Fusion and Hierarchical Variational Distillation. The former is comprised of two submodules, including Modality Reconstruction and Cross-Modality Augmented Transformer (CMA-Transformer), where Modality Reconstruction focuses on obtaining high-quality compressed representation of each modality, and CMA-Transformer adopts an asymmetric fusion strategy which treats one modality as the central modality and takes others as auxiliary modalities. The latter first designs a variational fusion network to fuse the fine-grained representations learned by CMA- Transformer into a coarse-grained representations. Then, it introduces a hierarchical distillation framework to maintain the consistency between modality representations with different granularities. Experiments on the IEMOCAP and MELD datasets demonstrate that our proposed model outperforms previous state-of-the-art baselines. Implementation codes can be available at <a class="link-external link-https" href="https://github.com/" rel="external noopener nofollow">this https URL</a> cjw-MER/CMATH.

AIMDiT: Modality Augmentation and Interaction via Multimodal Dimension Transformation for Emotion Recognition in Conversations

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

MM-DFN: Multimodal Dynamic Fusion Network for Emotion Recognition in Conversations

Fusing pairwise modalities for emotion recognition in conversations

Revisiting Disentanglement and Fusion on Modality and Context in Conversational Multimodal Emotion Recognition

CMATH: Cross-Modality Augmented Transformer with Hierarchical Variational Distillation for Multimodal Emotion Recognition in Conversation

Context-aware Multimodal Fusion for Emotion Recognition

Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning for Emotion Recognition in Conversation

Enhancing Emotion Recognition in Conversation through Emotional Cross-Modal Fusion and Inter-class Contrastive Learning

Emotion Recognition with Multimodal Transformer Fusion Framework Based on Acoustic and Lexical Information

A Contextual Attention Network for Multimodal Emotion Recognition in Conversation

A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations

Ada2I: Enhancing Modality Balance for Multimodal Conversational Emotion Recognition

Multimodal transformer augmented fusion for speech emotion recognition

Speaker-aware cognitive network with cross-modal attention for multimodal emotion recognition in conversation

MultiEMO: an Attention-Based Correlation-Aware Multimodal Fusion Framework for Emotion Recognition in Conversations.

Multimodal Fusion via Hypergraph Autoencoder and Contrastive Learning for Emotion Recognition in Conversation

Contextual and Cross-Modal Interaction for Multi-Modal Speech Emotion Recognition

MMDAG: Multimodal Directed Acyclic Graph Network for Emotion Recognition in Conversation