Abstract:In this paper, we address the problem of multimodal emotion recognition from multiple physiological signals. We demonstrate that a Transformer-based approach is suitable for this task. In addition, we present how such models may be pretrained in a multimodal scenario to improve emotion recognition performances. We evaluate the benefits of using multimodal inputs and pre-training with our approach on a state-ofthe-art dataset.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is multi - modal emotion recognition, especially recognizing emotions from multiple physiological signals. Specifically, the author focuses on how to use the Transformer model to process physiological signals (such as electrocardiogram (ECG) and electroencephalogram (EEG)) from wearable devices in order to improve the accuracy of emotion recognition. The main problems mentioned in the paper include: 1. **Insufficient data annotation**: There is a lack of sufficient annotated data in emotion recognition tasks to effectively train deep - learning models. To this end, the author explores unsupervised pre - training techniques to overcome this challenge. 2. **Multi - modal signal fusion**: How to effectively fuse physiological signals of different modalities to improve the performance of emotion recognition. The author proposes a Transformer - based late - fusion method, which pre - trains and fine - tunes single - modal models separately, and then combines the outputs of these models for the final emotion prediction. 3. **Applicability of the Transformer model**: Verify whether the Transformer model is suitable for processing physiological signals and whether its performance in emotion recognition tasks can be improved through pre - training techniques. The main contributions of the paper include: - Proposing a technique for recognizing emotions from multi - modal physiological signals using the Transformer model. - Describing a method for pre - training the Transformer model for recognizing emotions from multi - modal physiological signals. - Experimental results show that the multi - modal pre - training strategy can effectively improve the performance of emotion recognition. Through these studies, the author aims to provide new ideas and technical support for research in the field of emotion recognition, especially in using physiological signals for emotion recognition.

Emotion Recognition with Pre-Trained Transformers Using Multimodal Signals

Multimodal Transformer Fusion for Emotion Recognition: A Survey

Multi-Modal Emotion Recognition by Text, Speech and Video Using Pretrained Transformers

Multilevel Transformer For Multimodal Emotion Recognition

Transformer Based Multimodal Speech Emotion Recognition with Improved Neural Networks

Multimodal Neurophysiological Transformer for Emotion Recognition

A Unified Transformer-based Network for multimodal Emotion Recognition

Towards Learning a Joint Representation from Transformer in Multimodal Emotion Recognition

Transformer-Based Self-Supervised Learning for Emotion Recognition

A Unified Biosensor–Vision Multi-Modal Transformer network for emotion recognition

Facial Emotion Recognition with Inter-Modality-Attention-Transformer-Based Self-Supervised Learning

MMTrans-MT: A Framework for Multimodal Emotion Recognition Using Multitask Learning

Using transformers for multimodal emotion recognition: Taxonomies and state of the art review

A Proposal for Multimodal Emotion Recognition Using Aural Transformers and Action Units on RAVDESS Dataset

Multimodal Transformer Fusion for Continuous Emotion Recognition

Multimodal Speech Emotion Recognition Using Modality-specific Self-Supervised Frameworks

Multimodal transformer augmented fusion for speech emotion recognition

Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross Attention

Transformer-Based Multimodal Emotional Perception for Dynamic Facial Expression Recognition in the Wild

Emotion Recognition with Multimodal Transformer Fusion Framework Based on Acoustic and Lexical Information

Joint Multimodal Transformer for Emotion Recognition in the Wild