Abstract:In real-world human–computer interaction, the performance of multimodal emotion recognition models is inevitably affected by random modality feature missing. Thus, robust multimodal emotion recognition methods have attracted increasing attention. However, existing robust multimodal emotion recognition methods generally ignore the distributional gap between modalities. To address this issue, we propose a Twin Disentanglement Transformer Network with Hierarchical-Level Feature Reconstruction (TDTN-HLFR), which aims to enhance model robustness by reconstructing missing features from modality-common and modality-specific perspectives. The network comprises two primary learning processes: disentanglement learning and reconstruction learning. The former learns to efficiently decouple the original multimodal features into modality-specific and modality-common representations through a Disentanglement Transformer Network (DTN). Based on the former, the latter developed the TDTN-HLFR, which learns to reconstruct missing features from modality-common and modality-specific perspectives. In doing this, the TDTN-HLFR mitigates the impact of the distributional gap on the reconstruction of missing features. Extensive experiments are conducted on two multimodal continuous emotion recognition datasets: The Remote Collaborative and Affective (RECOLA) and the Ulm-Trier Social Stress Test (ULM-TSST) datasets. In terms of combined Concordance Correlation Coefficient (CCC) for valence and arousal prediction, our method delivers 0.0852 absolute increases on the RECOLA dataset and 0.0207 on the ULM-TSST dataset compared with the best baseline in the complete modality feature setting while delivering 0.0692 absolute increases on the RECOLA dataset and 0.0347 on the ULM-TSST dataset compared with the best baseline in the incomplete modality feature setting. These results demonstrate the potential of the TDTN-HLFR in real-world human–computer interaction scenarios.

Multiple Emotion Tagging for Multimedia Data by Exploiting High-Order Dependencies Among Emotions

Multiple Emotional Tagging of Multimedia Data by Exploiting Dependencies among Emotions.

Capturing Emotion Distribution for Multimedia Emotion Tagging

Capturing Dependencies among Labels and Features for Multiple Emotion Tagging of Multimedia Data.

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

Emotional Tagging Of Videos By Exploring Multiple Emotions' Coexistence

Exploiting Multi-Emotion Relations at Feature and Label Levels for Emotion Tagging

Exploiting Multi-Expression Dependences for Implicit Multi-Emotion Video Tagging.

Knowledge-Augmented Multimodal Deep Regression Bayesian Networks for Emotion Video Tagging

Implicit Video Multi-Emotion Tagging by Exploiting Multi-Expression Relations.

Content-Based Video Emotion Tagging Augmented by Users’ Multiple Physiological Responses

Bridging the Emotional Semantic Gap via Multimodal Relevance Estimation

Emotion Recognition Through Integrating EEG and Peripheral Signals

Multimodel emotion analysis in response to multimedia

Emotion recognition using multimodal deep learning in multiple psychophysiological signals and video

Multi-modal Multi-label Emotion Recognition with Heterogeneous Hierarchical Message Passing

A twin disentanglement Transformer Network with Hierarchical-Level Feature Reconstruction for robust multimodal emotion recognition

EffMulti: Efficiently Modeling Complex Multimodal Interactions for Emotion Analysis

Implicit Hybrid Video Emotion Tagging By Integrating Video Content And Users' Multiple Physiological Responses

InterMulti:Multi-view Multimodal Interactions with Text-dominated Hierarchical High-order Fusion for Emotion Analysis

A Multimodal Deep Regression Bayesian Network For Affective Video Content Analyses