Abstract:Electroencephalography (EEG)-based emotion classification during music listening has gained increasing attention nowadays due to its promise of potential applications such as musical affective brain-computer interface (ABCI), neuromarketing, music therapy, and implicit multimedia tagging and triggering. However, music is an ecologically valid and complex stimulus that conveys certain emotions to listeners through compositions of musical elements. Using solely EEG signals to distinguish emotions remained challenging. This study aimed to assess the applicability of a multimodal approach by leveraging the EEG dynamics and acoustic characteristics of musical contents for the classification of emotional valence and arousal. To this end, this study adopted machine-learning methods to systematically elucidate the roles of the EEG and music modalities in the emotion modeling. The empirical results suggested that when whole-head EEG signals were available, the inclusion of musical contents did not improve the classification performance. The obtained performance of 74~76% using solely EEG modality was statistically comparable to that using the multimodality approach. However, if EEG dynamics were only available from a small set of electrodes (likely the case in real-life applications), the music modality would play a complementary role and augment the EEG results from around 61-67% in valence classification and from around 58-67% in arousal classification. The musical timber appeared to replace less-discriminative EEG features and led to improvements in both valence and arousal classification, whereas musical loudness was contributed specifically to the arousal classification. The present study not only provided principles for constructing an EEG-based multimodal approach, but also revealed the fundamental insights into the interplay of the brain activity and musical contents in emotion modeling.

Bi-Modal Deep Boltzmann Machine Based Musical Emotion Classification

Automatic Music Emotion Classification Using a New Classification Algorithm

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Human-centric Music Medical Therapy Exploration System

Automatic Music Mood Classification by Learning Cross-Media Relevance Between Audio and Lyrics

Boosting for Multi-Modal Music Emotion Classification.

Graph-Based Multimodal Music Mood Classification in Discriminative Latent Space.

Real-Time Human-Music Emotional Interaction Based on Deep Learning and Multimodal Sentiment Analysis

Emotional Recognition For Chime Bell Music

User-Adaptive Music Emotion Recognition

A Deep Bidirectional Long Short-Term Memory Based Multi-Scale Approach for Music Dynamic Emotion Prediction

Fusion of electroencephalographic dynamics and musical contents for estimating emotional responses in music listening

Enhancing Music Mood Recognition with LLMs and Audio Signal Processing: A Multimodal Approach

MMD-MII Model: A Multilayered Analysis and Multimodal Integration Interaction Approach Revolutionizing Music Emotion Classification

Music emotion recognition based on temporal convolutional attention network using EEG

Real-time Human-Music Emotional Interaction Based on Multimodal Analysis

Multimodal Music Mood Classification by Fusion of Audio and Lyrics.

Digital Empirical Research of Influencing Factors of Musical Emotion Classification Based on Pleasure-Arousal Musical Emotion Fuzzy Model

Multimodal Music Emotion Recognition with Hierarchical Cross-Modal Attention Network

Research on Music Emotional Expression Based on Reinforcement Learning and Multimodal Information

Music-evoked Emotion Classification Using EEG Correlation-Based Information