Abstract:Multimodal emotion recognition in conversation (MER) aims to accurately identify emotions in conversational utterances by integrating multimodal information. Previous methods usually treat multimodal information as equal quality and employ symmetric architectures to conduct multimodal fusion. However, in reality, the quality of different modalities usually varies considerably, and utilizing a symmetric architecture is difficult to accurately recognize conversational emotions when dealing with uneven modal information. Furthermore, fusing multi-modality information in a single granularity may fail to adequately integrate modal information, exacerbating the inaccuracy in emotion recognition. In this paper, we propose a novel Cross-Modality Augmented Transformer with Hierarchical Variational Distillation, called CMATH, which consists of two major components, i.e., Multimodal Interaction Fusion and Hierarchical Variational Distillation. The former is comprised of two submodules, including Modality Reconstruction and Cross-Modality Augmented Transformer (CMA-Transformer), where Modality Reconstruction focuses on obtaining high-quality compressed representation of each modality, and CMA-Transformer adopts an asymmetric fusion strategy which treats one modality as the central modality and takes others as auxiliary modalities. The latter first designs a variational fusion network to fuse the fine-grained representations learned by CMA- Transformer into a coarse-grained representations. Then, it introduces a hierarchical distillation framework to maintain the consistency between modality representations with different granularities. Experiments on the IEMOCAP and MELD datasets demonstrate that our proposed model outperforms previous state-of-the-art baselines. Implementation codes can be available at <a class="link-external link-https" href="https://github.com/" rel="external noopener nofollow">this https URL</a> cjw-MER/CMATH.

MutualFormer: Multi-modal Representation Learning via Cross-Diffusion Attention

X-Gacmn: An X-Shaped Generative Adversarial Cross-Modal Network With Hypersphere Embedding

Mutually Beneficial Transformer for Multimodal Data Fusion

Improved Transformer with Multi-Head Dense Collaboration

Improving Transformers with Dynamically Composable Multi-Head Attention

Multimodal Token Fusion for Vision Transformers

MTFormer: Multi-task Learning via Transformer and Cross-Task Reasoning.

MAFormer: A transformer network with multi-scale attention fusion for visual recognition

Multi-hop neighbor fusion enhanced hierarchical transformer for multi-modal knowledge graph completion

Multi-scale Cooperative Multimodal Transformers for Multimodal Sentiment Analysis in Videos

Multimodal Transformer for Unaligned Multimodal Language Sequences

Towards Multi-modal Transformers in Federated Learning

Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goals

Multi-label adversarial fine-grained cross-modal retrieval

Multi-Modal Representation via Contrastive Learning with Attention Bottleneck Fusion and Attentive Statistics Features

Meta-Transformer: A Unified Framework for Multimodal Learning

TripleFormer: improving transformer-based image classification method using multiple self-attention inputs

Transformer-based Multi-Modal Learning for Multi Label Remote Sensing Image Classification

CAVER: Cross-Modal View-Mixed Transformer for Bi-Modal Salient Object Detection

Token-disentangling Mutual Transformer for multimodal emotion recognition

CMATH: Cross-Modality Augmented Transformer with Hierarchical Variational Distillation for Multimodal Emotion Recognition in Conversation