Abstract:Considerable attention has been paid to physiological signal-based emotion recognition in the field of affective computing. For reliability and user-friendly acquisition, electrodermal activity (EDA) has a great advantage in practical applications. However, EDA-based emotion recognition with large-scale subjects is still a tough problem. The traditional well-designed classifiers with hand-crafted features produce poorer results because of their limited representation abilities. And the deep learning models with auto feature extraction suffer the overfitting drop-off because of large-scale individual differences. Since music has a strong correlation with human emotion, static music can be involved as the external benchmark to constrain various dynamic EDA signals. In this article, we make an attempt by fusing the subject’s individual EDA features and the external evoked music features. And we propose an end-to-end multimodal framework, the one-dimensional residual temporal and channel attention network (RTCAN-1D). For EDA features, the channel-temporal attention mechanism for EDA-based emotion recognition is first involved in mine the temporal and channel-wise dynamic and steady features. The comparisons with single EDA-based SOTA models on DEAP and AMIGOS datasets prove the effectiveness of RTCAN-1D to mine EDA features. For music features, we simply process the music signal with the open-source toolkit openSMILE to obtain external feature vectors. We conducted systematic and extensive evaluations. The experiments on the current largest music emotion dataset PMEmo validate that the fusion of EDA and music is a reliable and efficient solution for large-scale emotion recognition.

Multi-scale Context Based Attention for Dynamic Music Emotion Prediction

A Deep Bidirectional Long Short-Term Memory Based Multi-Scale Approach for Music Dynamic Emotion Prediction

DBLSTM-based Multi-Scale Fusion for Dynamic Emotion Prediction in Music.

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Multi-Scale Approaches to the MediaEval 2015 "emotion in Music" Task.

Svr Based Double-Scale Regression For Dynamic Emotion Prediction In Music

Music emotion prediction based on multidimensional feature fusion

Continuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network

User-Adaptive Music Emotion Recognition

Multi-Dimensional Music Emotion Recognition Incorporating Convolutional Neural Networks and Plutchik's Emotion Wheel.

Dynamic Music Emotion Recognition Based on CNN-BiLSTM

Multi-scale Temporal Modeling for Dimensional Emotion Recognition in Video

Music emotion recognition based on temporal convolutional attention network using EEG

Multimodal Music Emotion Recognition with Hierarchical Cross-Modal Attention Network

Dimensional Music Emotion Recognition by Machine Learning

Multi-modal Emotion Recognition with Temporal-Band Attention Based on LSTM-RNN.

A Multimodal Framework for Large-Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Personalized Dynamic Music Emotion Recognition with Dual-Scale Attention-Based Meta-Learning

ADFF: Attention Based Deep Feature Fusion Approach for Music Emotion Recognition

Dimensional Music Emotion Recognition by Valence-Arousal Regression.

Research on Music Emotional Expression Based on Reinforcement Learning and Multimodal Information