Abstract:In real-world human–computer interaction, the performance of multimodal emotion recognition models is inevitably affected by random modality feature missing. Thus, robust multimodal emotion recognition methods have attracted increasing attention. However, existing robust multimodal emotion recognition methods generally ignore the distributional gap between modalities. To address this issue, we propose a Twin Disentanglement Transformer Network with Hierarchical-Level Feature Reconstruction (TDTN-HLFR), which aims to enhance model robustness by reconstructing missing features from modality-common and modality-specific perspectives. The network comprises two primary learning processes: disentanglement learning and reconstruction learning. The former learns to efficiently decouple the original multimodal features into modality-specific and modality-common representations through a Disentanglement Transformer Network (DTN). Based on the former, the latter developed the TDTN-HLFR, which learns to reconstruct missing features from modality-common and modality-specific perspectives. In doing this, the TDTN-HLFR mitigates the impact of the distributional gap on the reconstruction of missing features. Extensive experiments are conducted on two multimodal continuous emotion recognition datasets: The Remote Collaborative and Affective (RECOLA) and the Ulm-Trier Social Stress Test (ULM-TSST) datasets. In terms of combined Concordance Correlation Coefficient (CCC) for valence and arousal prediction, our method delivers 0.0852 absolute increases on the RECOLA dataset and 0.0207 on the ULM-TSST dataset compared with the best baseline in the complete modality feature setting while delivering 0.0692 absolute increases on the RECOLA dataset and 0.0347 on the ULM-TSST dataset compared with the best baseline in the incomplete modality feature setting. These results demonstrate the potential of the TDTN-HLFR in real-world human–computer interaction scenarios.

A Simple and Interactive Transformer for Fine-Grained Emotion Detection

Emotion-Aware Transformer Encoder for Empathetic Dialogue Generation

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

Multilevel Transformer For Multimodal Emotion Recognition

Transformer-Based Multimodal Emotional Perception for Dynamic Facial Expression Recognition in the Wild

A Hierarchical Transformer with Speaker Modeling for Emotion Recognition in Conversation

Modality-collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognition

Emotion Recognition Using Transformers with Masked Learning

Emotion Transformer Fusion - Complementary Representation Properties of EEG and Eye Movements on Recognizing Anger and Surprise.

Hierarchical Transformer Network for Utterance-Level Emotion Recognition

A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations

EmT: A Novel Transformer for Generalized Cross-subject EEG Emotion Recognition

Temporal-spatial Representation Learning Transformer for EEG-based Emotion Recognition

Short and Long Range Relation Based Spatio-Temporal Transformer for Micro-Expression Recognition

Multi-Scale Temporal Transformer For Speech Emotion Recognition

TDFNet: Transformer-Based Deep-Scale Fusion Network for Multimodal Emotion Recognition

Learning Modality-Fused Representation Based on Transformer for Emotion Analysis

An End-to-End Transformer with Progressive Tri-Modal Attention for Multi-modal Emotion Recognition.

A twin disentanglement Transformer Network with Hierarchical-Level Feature Reconstruction for robust multimodal emotion recognition

SS-Trans (Single-Stream Transformer for Multimodal Sentiment Analysis and Emotion Recognition): The Emotion Whisperer—A Single-Stream Transformer for Multimodal Sentiment Analysis