Abstract:With the advancement of human-computer interaction, the role of emotion recognition has become increasingly significant. Emotion recognition technology provides practical benefits across various industries, including user experience enhancement, education, and organizational productivity. For instance, in educational settings, it enables real-time understanding of students' emotional states, facilitating tailored feedback. In workplaces, monitoring employees' emotions can contribute to improved job performance and satisfaction. Recently, emotion recognition has also gained attention in media applications such as automated movie dubbing, where it enhances the naturalness of dubbed performances by synchronizing emotional expression in both audio and visuals. Consequently, multimodal emotion recognition research, which integrates text, speech, and video data, has gained momentum in diverse fields. In this study, we propose an emotion recognition approach that combines text and speech data, specifically incorporating the characteristics of the Korean language. For text data, we utilize KoELECTRA to generate embeddings, and for speech data, we extract features using HuBERT embeddings. The proposed multimodal transformer model processes text and speech data independently, subsequently learning interactions between the two modalities through a Cross-Modal Attention mechanism. This approach effectively combines complementary information from text and speech, enhancing the accuracy of emotion recognition. Our experimental results demonstrate that the proposed model surpasses single-modality models, achieving a high accuracy of 77.01% and an F1-Score of 0.7703 in emotion classification. This study contributes to the advancement of emotion recognition technology by integrating diverse language and modality data, suggesting the potential for further improvements through the inclusion of additional modalities in future work.

Bi-Modal Bi-Task Emotion Recognition Based on Transformer Architecture

Multilevel Transformer For Multimodal Emotion Recognition

Transformer Based Multimodal Speech Emotion Recognition with Improved Neural Networks

Multi-Modal Emotion Recognition by Text, Speech and Video Using Pretrained Transformers

Multimodal Transformer with Learnable Frontend and Self Attention for Emotion Recognition

Multimodal transformer augmented fusion for speech emotion recognition

Topic and Style-aware Transformer for Multimodal Emotion Recognition

Transformer-Based Multimodal Emotional Perception for Dynamic Facial Expression Recognition in the Wild

A Unified Transformer-based Network for multimodal Emotion Recognition

Multimodal Speech Emotion Recognition Using Modality-specific Self-Supervised Frameworks

Multi-head attention fusion networks for multi-modal speech emotion recognition

Multi-Label Multimodal Emotion Recognition With Transformer-Based Fusion and Emotion-Level Representation Learning

Modality-collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognition

SS-Trans (Single-Stream Transformer for Multimodal Sentiment Analysis and Emotion Recognition): The Emotion Whisperer—A Single-Stream Transformer for Multimodal Sentiment Analysis

Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross Attention

Multi-Modal Emotion Recognition Based on Wavelet Transform and BERT-RoBERTa: An Innovative Approach Combining Enhanced BiLSTM and Focus Loss Function

KoHMT: A Multimodal Emotion Recognition Model Integrating KoELECTRA, HuBERT with Multimodal Transformer

Multimodal Transformer Fusion for Continuous Emotion Recognition

Towards Learning a Joint Representation from Transformer in Multimodal Emotion Recognition

TMFER: Multimodal Fusion Emotion Recognition Algorithm Based on Transformer

Modulated Fusion using Transformer for Linguistic-Acoustic Emotion Recognition