Abstract:In recent years, speech emotion recognition (SER) increasingly attracts attention since it is a key component of intelligent human-computer interaction and sophisticated dialog systems. To obtain more abundant emotional information, a great number of studies in SER pay attention to the multimodal systems which utilize other modalities such as text and facial expression to assist the speech emotion recognition. However, it is difficult to structure a fusion mechanism which can selectively extract abundant emotion-related features from different modalities. To tackle this issue, we develop a multimodal speech emotion recognition model based on multi-scale MFCCs and multi-view attention mechanism, which is able to extract abundant audio emotional features and comprehensively fuse emotion-related features from four aspects (i.e., audio self-attention, textual self-attention, audio attention based on textual content, and textual attention based on audio content). Under different audio input conditions and attention configurations, it can be observed that the best emotion recognition accuracy can be achieved by jointly utilizing four attention modules and three different scales of MFCCs. In addition, based on multi-task learning, we regard the gender recognition as an auxiliary task to learn gender information. To further improve the accuracy of emotion recognition, a joint loss function based on softmax cross-entropy loss and center loss is used. The experiments are conducted on two different datasets (IEMOCAP and MSP-IMPROV). The experimental results demonstrate that the proposed model outperforms the previous models on IEMOCAP dataset, while it obtains the competitive performance on MSP-IMPROV dataset.

Speech Emotion Recognition in Dyadic Dialogues with Attentive Interaction Modeling

Deep Spectrum Feature Representations for Speech Emotion Recognition

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Self-attention Transfer Networks for Speech Emotion Recognition

A Contextual Attention Network for Multimodal Emotion Recognition in Conversation

Self-adaptive Context and Modal-interaction Modeling For Multimodal Emotion Recognition

Multimodal Dimensional and Continuous Emotion Recognition in Dyadic Video Interactions.

Emotion Recognition via Environmental Context and Human Body

Conversational Emotion Analysis via Attention Mechanisms

DialogueEIN: Emotion Interaction Network for Dialogue Affective Analysis.

A Self-Attentive Emotion Recognition Network

Emotion embedding framework with emotional self-attention mechanism for speaker recognition

Multi-Modal Multi-Cultural Dimensional Continues Emotion Recognition In Dyadic Interactions

Context-aware Multimodal Fusion for Emotion Recognition

Speech Emotion Recognition Using Multihead Attention in Both Time and Feature Dimensions

Multimodal Speech Emotion Recognition Based on Multi-Scale MFCCs and Multi-View Attention Mechanism

AIMDiT: Modality Augmentation and Interaction via Multimodal Dimension Transformation for Emotion Recognition in Conversations

Speaker-aware cognitive network with cross-modal attention for multimodal emotion recognition in conversation

Adapted Dynamic Memory Network for Emotion Recognition in Conversation

Language-guided Multi-modal Emotional Mimicry Intensity Estimation

Dynamic Interactive Multiview Memory Network for Emotion Recognition in Conversation