Abstract:Sentiment analysis is an important research field aiming to extract and fuse sentimental information from human utterances. Due to the diversity of human sentiment, analyzing from multiple modalities is usually more accurate than from a single modality. To complement the information between related modalities, one effective approach is performing cross-modality interactions. Recently, Transformer-based frameworks have shown a strong ability to capture long-range dependencies, leading to the introduction of several Transformer-based approaches for multimodal processing. However, due to the built-in attention mechanism of the Transformers, only two modalities can be engaged at once. As a result, the complementary information flow in these Transformer-based techniques is partial and constrained. To mitigate this, we propose, TensorFormer, a tensor-based multimodal Transformer framework that takes into account all relevant modalities for interactions. More precisely, we first construct a tensor utilizing the features extracted from each modality, assuming one modality is the target while the remaining tensors serve as the sources. We can generate the corresponding interacted features by calculating source-target attention. This strategy interacts with all involved modalities and generates complementing global information. Experiments on multimodal sentiment analysis benchmark datasets demonstrated the effectiveness of TensorFormer. In addition, we also evaluate TensorFormer in another related area: depression detection and the results reveal significant improvements when compared to other state-of-the-art methods.

Multimodal Transformer Fusion for Emotion Recognition: A Survey

Using transformers for multimodal emotion recognition: Taxonomies and state of the art review

TensorFormer: A Tensor-Based Multimodal Transformer for Multimodal Sentiment Analysis and Depression Detection

Multimodal transformer augmented fusion for speech emotion recognition

Multimodal Transformer Fusion for Continuous Emotion Recognition

Multilevel Transformer For Multimodal Emotion Recognition

Emotion Recognition with Pre-Trained Transformers Using Multimodal Signals

Multi-Modal Emotion Recognition by Text, Speech and Video Using Pretrained Transformers

TMFER: Multimodal Fusion Emotion Recognition Algorithm Based on Transformer

Multi-Label Multimodal Emotion Recognition With Transformer-Based Fusion and Emotion-Level Representation Learning

Research on Multimodal Emotion Recognition Based on Fusion of Electroencephalogram and Electrooculography

Emotion Recognition with Multimodal Transformer Fusion Framework Based on Acoustic and Lexical Information

Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross Attention

Learning Modality-Fused Representation Based on Transformer for Emotion Analysis

Transformer Based Multimodal Speech Emotion Recognition with Improved Neural Networks

Multimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion

Multimodal Neurophysiological Transformer for Emotion Recognition

Low Rank Fusion based Transformers for Multimodal Sequences

Emotion-Aware Transformer Encoder for Empathetic Dialogue Generation

Modulated Fusion using Transformer for Linguistic-Acoustic Emotion Recognition

A Unified Transformer-based Network for multimodal Emotion Recognition