Abstract:Multimodal emotion recognition in conversation (MER) aims to accurately identify emotions in conversational utterances by integrating multimodal information. Previous methods usually treat multimodal information as equal quality and employ symmetric architectures to conduct multimodal fusion. However, in reality, the quality of different modalities usually varies considerably, and utilizing a symmetric architecture is difficult to accurately recognize conversational emotions when dealing with uneven modal information. Furthermore, fusing multi-modality information in a single granularity may fail to adequately integrate modal information, exacerbating the inaccuracy in emotion recognition. In this paper, we propose a novel Cross-Modality Augmented Transformer with Hierarchical Variational Distillation, called CMATH, which consists of two major components, i.e., Multimodal Interaction Fusion and Hierarchical Variational Distillation. The former is comprised of two submodules, including Modality Reconstruction and Cross-Modality Augmented Transformer (CMA-Transformer), where Modality Reconstruction focuses on obtaining high-quality compressed representation of each modality, and CMA-Transformer adopts an asymmetric fusion strategy which treats one modality as the central modality and takes others as auxiliary modalities. The latter first designs a variational fusion network to fuse the fine-grained representations learned by CMA- Transformer into a coarse-grained representations. Then, it introduces a hierarchical distillation framework to maintain the consistency between modality representations with different granularities. Experiments on the IEMOCAP and MELD datasets demonstrate that our proposed model outperforms previous state-of-the-art baselines. Implementation codes can be available at <a class="link-external link-https" href="https://github.com/" rel="external noopener nofollow">this https URL</a> cjw-MER/CMATH.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to more effectively fuse modal information of different qualities in multi - modal emotion recognition to improve the accuracy of dialogue emotion recognition. Specifically, the paper points out that current methods usually assume that different modal information has the same quality and adopt a symmetric architecture for multi - modal fusion. However, in practical applications, there are often significant differences in the information quality of different modalities, and this symmetric fusion strategy is difficult to accurately recognize emotions in dialogue when dealing with uneven modal information. In addition, the single - granularity information fusion method may not fully integrate modal information, thus exacerbating the inaccuracy of emotion recognition. To overcome these problems, the paper proposes a new method - **CMATH (Cross - Modality Augmented Transformer with Hierarchical Variational Distillation)**. This method mainly consists of two modules: 1. **Multimodal Interaction Fusion Module**: - **Modality Reconstruction**: Down - sample and up - sample the original representations of different modalities through 1D convolutional layers to obtain high - quality compressed representations, and optimize these representations through reconstruction loss. - **Cross - Modality Augmented Transformer (CMA - Transformer)**: Adopt an asymmetric fusion strategy, regard one modality as the central modality and other modalities as auxiliary modalities, and enhance the representation of the central modality through the cross - attention mechanism and the feed - forward network. 2. **Hierarchical Variational Distillation Module**: - **Variational Fusion Network**: Fuse the fine - grained representations generated by the CMA - Transformer into coarse - grained representations, learn the Gaussian distribution parameters of each modality through a multi - layer perceptron, and sample using the reparameterization technique. - **Hierarchical Distillation**: Maintain the consistency between modal representations of different granularities through low - level distillation and high - level distillation strategies. Low - level distillation minimizes the difference between the teacher model and the student model in the semantic representation space, while high - level distillation minimizes the KL divergence in the decision representation space. Through these innovations, CMATH can more accurately recognize emotions in dialogue when dealing with multi - modal information of different qualities. Experimental results show that CMATH significantly outperforms existing baseline methods on the IEMOCAP and MELD datasets and achieves state - of - the - art performance.

CMATH: Cross-Modality Augmented Transformer with Hierarchical Variational Distillation for Multimodal Emotion Recognition in Conversation

A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations

Multimodal Fusion via Hypergraph Autoencoder and Contrastive Learning for Emotion Recognition in Conversation

Multimodal Emotion Recognition Based on Cascaded Multichannel and Hierarchical Fusion

Modality-collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognition

Decoupled Multimodal Distilling for Emotion Recognition

Multimodal transformer augmented fusion for speech emotion recognition

AIMDiT: Modality Augmentation and Interaction via Multimodal Dimension Transformation for Emotion Recognition in Conversations

MMGCN: Multimodal Fusion Via Deep Graph Convolution Network for Emotion Recognition in Conversation

Multilevel Transformer For Multimodal Emotion Recognition

Multi-modal emotion recognition using tensor decomposition fusion and self-supervised multi-tasking

Multimodal Prompt Transformer with Hybrid Contrastive Learning for Emotion Recognition in Conversation

First-order Multi-label Learning with Cross-modal Interactions for Multimodal Emotion Recognition

A Multi-Level Circulant Cross-Modal Transformer for Multimodal Speech Emotion Recognition

TMFER: Multimodal Fusion Emotion Recognition Algorithm Based on Transformer

Contextual and Cross-Modal Interaction for Multi-Modal Speech Emotion Recognition

Muti-modal Emotion Recognition Via Hierarchical Knowledge Distillation

Joint Multimodal Transformer for Emotion Recognition in the Wild

LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences

Token-disentangling Mutual Transformer for multimodal emotion recognition