Abstract:Electroencephalography (EEG)-based emotion classification during music listening has gained increasing attention nowadays due to its promise of potential applications such as musical affective brain-computer interface (ABCI), neuromarketing, music therapy, and implicit multimedia tagging and triggering. However, music is an ecologically valid and complex stimulus that conveys certain emotions to listeners through compositions of musical elements. Using solely EEG signals to distinguish emotions remained challenging. This study aimed to assess the applicability of a multimodal approach by leveraging the EEG dynamics and acoustic characteristics of musical contents for the classification of emotional valence and arousal. To this end, this study adopted machine-learning methods to systematically elucidate the roles of the EEG and music modalities in the emotion modeling. The empirical results suggested that when whole-head EEG signals were available, the inclusion of musical contents did not improve the classification performance. The obtained performance of 74~76% using solely EEG modality was statistically comparable to that using the multimodality approach. However, if EEG dynamics were only available from a small set of electrodes (likely the case in real-life applications), the music modality would play a complementary role and augment the EEG results from around 61-67% in valence classification and from around 58-67% in arousal classification. The musical timber appeared to replace less-discriminative EEG features and led to improvements in both valence and arousal classification, whereas musical loudness was contributed specifically to the arousal classification. The present study not only provided principles for constructing an EEG-based multimodal approach, but also revealed the fundamental insights into the interplay of the brain activity and musical contents in emotion modeling.

Effects of Individual Factors on Perceived Emotion and Felt Emotion of Music: Based on Machine Learning Methods

Using Psychophysiologicalmeasures to Recognize Personalmusic Emotional Experience

Human-centric Music Medical Therapy Exploration System

Digital Empirical Research of Influencing Factors of Musical Emotion Classification Based on Pleasure-Arousal Musical Emotion Fuzzy Model

Physiological State Can Help Predict the Perceived Emotion of Music: Evidence from ECG and EDA Signals

Image–Music Synesthesia-Aware Learning Based on Emotional Similarity Recognition

Study on Linguistic Computing for Music Emotion

Fusion of electroencephalographic dynamics and musical contents for estimating emotional responses in music listening

User-Adaptive Music Emotion Recognition

Music's Affective Computing Model Based on Fuzzy Logic

Music Emotion Research Based on Reinforcement Learning and Multimodal Information

Symbolic & Acoustic: Multi-domain Music Emotion Modeling for Instrumental Music

Exploring User Emotion in Microblogs for Music Recommendation.

A survey of music emotion recognition

Real-Time Human-Music Emotional Interaction Based on Deep Learning and Multimodal Sentiment Analysis

Emotional Recognition For Chime Bell Music

Automatic ECG-Based Emotion Recognition in Music Listening

EEG Emotion Recognition Applied to the Effect Analysis of Music on Emotion Changes in Psychological Healthcare

Real-time Human-Music Emotional Interaction Based on Multimodal Analysis

Analysis and recognition of a novel experimental paradigm for musical emotion brain-computer interface

An Emotion-Based Personalized Music Recommendation Framework for Emotion Improvement