PKU-AIPL' Solution for MediaEval 2015 Emotion in Music Task.

Kang Cai,Wanyi Yang,Yao Cheng,Deshun Yang,Xiaoou Chen

2015-01-01

Abstract:In this paper, we describe the PKU-AIPL Team solution of Emotion in Music task in MediaEval benchmarking campaign 2015. We extracted and designed several sets of features and used continuous conditional random eld(CCRF) for dynamic emotion characterization task.

What problem does this paper attempt to address?

Beatsens' Solution for MediaEval 2014 Emotion in Music Task.

Wanyi Yang,Kang Cai,Bin Wu,Ying Wang,Xiaoou Chen,Deshun Yang,Andrew Horner

2014-01-01

Abstract:In this paper, we describe the Beatsens Team solution of Emotion in Music task in MediaEval benchmarking campaign 2014. We extracted and designed several sets of features and used continuous conditional random eld(CCRF) for dynamic emotion characterization task. The best runs for Pearson correlation are 0:23 0:56 and 0:12 0:55 of valence and arousal respectively, for RMSE are 0:12 0:06 and 0:09 0:05.
A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Guanghao Yin,Shouqian Sun,Dian Yu,Dejian Li,Kejun Zhang

DOI: https://doi.org/10.1145/3490686

IF: 4.094

2022-01-01

ACM Transactions on Multimedia Computing Communications and Applications

Abstract:Considerable attention has been paid for physiological signal-based emotion recognition in field of affective computing. For the reliability and user friendly acquisition, Electrodermal Activity (EDA) has great advantage in practical applications. However, the EDA-based emotion recognition with hundreds of subjects still lacks effective solution. In this paper, our work makes an attempt to fuse the subject individual EDA features and the external evoked music features. And we propose an end-to-end multimodal framework, the 1-dimensional residual temporal and channel attention network (RTCAN-1D). For EDA features, the novel convex optimization-based EDA (CvxEDA) method is applied to decompose EDA signals into pahsic and tonic signals for mining the dynamic and steady features. The channel-temporal attention mechanism for EDA-based emotion recognition is firstly involved to improve the temporal- and channel-wise representation. For music features, we process the music signal with the open source toolkit openSMILE to obtain external feature vectors. The individual emotion features from EDA signals and external emotion benchmarks from music are fused in the classifing layers. We have conducted systematic comparisons on three multimodal datasets (PMEmo, DEAP, AMIGOS) for 2-classes valance/arousal emotion recognition. Our proposed RTCAN-1D outperforms the existing state-of-the-art models, which also validate that our work provides an reliable and efficient solution for large scale emotion recognition. Our code has been released at this https URL.
Learning Music Emotion Primitives via Supervised Dynamic Clustering.

Yang Liu,Yan Liu,Xiang Zhang,Gong Chen,Kejun Zhang

DOI: https://doi.org/10.1145/2964284.2967215

2016-01-01

Abstract:This paper explores a fundamental problem in music emotion analysis, i.e., how to segment the music sequence into a set of basic emotive units, which are named as emotion primitives. Current works on music emotion analysis are mainly based on the fixed-length music segments, which often leads to the difficulty of accurate emotion recognition. Short music segment, such as an individual music frame, may fail to evoke emotion response. Long music segment, such as an entire song, may convey various emotions over time. Moreover, the minimum length of music segment varies depending on the types of the emotions. To address these problems, we propose a novel method dubbed supervised dynamic clustering (SDC) to automatically decompose the music sequence into meaningful segments with various lengths. First, the music sequence is represented by a set of music frames. Then, the music frames are clustered according to the valence-arousal values in the emotion space. The clustering results are used to initialize the music segmentation. After that, a dynamic programming scheme is employed to jointly optimize the subsequent segmentation and grouping in the music feature space. Experimental results on standard dataset show both the effectiveness and the rationality of the proposed method.
Multi-Scale Approaches to the MediaEval 2015 "emotion in Music" Task.

Mingxing Xu,Xinxing Li,Haishu Xianyu,Jiashen Tian,Fanhang Meng,Wenxiao Chen

2015-01-01

Abstract:The goal of the “Emotion in Music” task in MediaEval 2015 is to automatically estimate the emotions expressed by music (in terms of Arousal and Valence) in a time-continuous fashion. In this paper, considering the high context correlation among the music feature sequence, we study several multiscale approaches at different levels, including acoustic feature learning with Deep Brief Networks (DBNs) followed a modified Autoencoder (AE), bi-directional Long-Short Term Memory Recurrent Neural Networks (BLSTM-RNNs) based multi-scale regression fusion with Extreme Learning Machine (ELM), and hierarchical prediction with Support Vector Regression (SVR). The evaluation performances of all runs submitted are significantly better than the baseline provided by the organizers, illustrating the effectiveness of the proposed approaches.
Emotion Recognition in Videos via Fusing Multimodal Features.

Shizhe Chen,Yujie Dian,Xinrui Li,Xiaozhu Lin,Qin Jin,Haibo Liu,Li Lu

DOI: https://doi.org/10.1007/978-981-10-3005-5_52

2016-01-01

Abstract:Emotion recognition is a challenging task with a wide range of applications. In this paper, we present our system in the CCPR 2016 multimodal emotion recognition challenge. Multimodal features from acoustic signals, facial expressions and speech contents are extracted to recognize the emotion of the character in the video. Among them the facial CNN feature is the most discriminative feature for emotion recognition. We train SVM and random forest classifiers based on each type of features and utilize early and late fusion to combine the different modality features. To deal with the data unbalance issue, we propose to adapt the probability thresholds for each emotion class. The macro precision of our best multimodal fusion system achieves 50.34% on the testing set, which significantly outperforms the baseline of 30.63 %.
Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

Man Hao,Wei-Hua Cao,Zhen-Tao Liu,Min Wu,Peng Xiao

DOI: https://doi.org/10.1016/j.neucom.2020.01.048

IF: 6

2020-01-01

Neurocomputing

Abstract:An ensemble visual-audio emotion recognition framework is proposed based on multi-task and blending learning with multiple features in this paper. To solve the problem that existing features can not accurately identify different emotions, we extract two kinds features, i. e., Interspeech 2010 and deep features for audio data, LBP and deep features for visual data, with the intent to accurately identify different emotions by using different features. Owing to the diversity of features, SVM classifiers and CNN are designed for manual features, i.e., Interspeech 2010 features and local LBP features, and deep features, through which four sub-models are obtained. Finally, the blending ensemble algorithm is used to fuse sub-models to improve the recognition performance of visual-audio emotion recognition. In addition, multi-task learning is applied in the CNN model for deep features, which can predict multiple tasks at the same time with fewer parameters and improve the sensitivity of the single recognition model to user’s emotion by sharing information between different tasks. Experiments are performed using eNTERFACCE database, from which the results indicate that the recognition of multi-task CNN increased by 3% and 2% on average over CNN model in speaker-independent and speaker-dependent experiments, respectively. And emotion recognition accuracy of visual-audio by our method reaches 81.36% and 78.42% in speaker-independent and speaker-dependent experiments, respectively, which maintain higher performance than some state-of-the-art works.
Solution for Emotion Prediction Competition of Workshop on Emotionally and Culturally Intelligent AI

Shengdong Xu,Zhouyang Chi,Yang Yang

2024-03-31

Abstract:This report provide a detailed description of the method that we explored and proposed in the WECIA Emotion Prediction Competition (EPC), which predicts a person's emotion through an artistic work with a comment. The dataset of this competition is ArtELingo, designed to encourage work on diversity across languages and cultures. The dataset has two main challenges, namely modal imbalance problem and language-cultural differences problem. In order to address this issue, we propose a simple yet effective approach called single-multi modal with Emotion-Cultural specific prompt(ECSP), which focuses on using the single modal message to enhance the performance of multimodal models and a well-designed prompt to reduce cultural differences problem. To clarify, our approach contains two main blocks: (1)XLM-R\cite{conneau2019unsupervised} based unimodal model and X$^2$-VLM\cite{zeng2022x} based multimodal model (2) Emotion-Cultural specific prompt. Our approach ranked first in the final test with a score of 0.627.

Artificial Intelligence
MediaEval 2014: THU-HCSIL Approach to Emotion in Music Task Using Multi-level Regression.

Yuchao Fan,Mingxing Xu

2014-01-01

Abstract:This working notes paper describes the system proposed by THU-HCSIL team for dynamic music emotion recognition. The procedure is divided into two module - feature extraction and regression. Both feature selection and feature combination are used to form the nal THU feature set. In regression module, a Booster-based Multi-level Regression method is presented, which outperforms the baseline significantly on test data in RMSE metric for dynamic task.
Multimodal Feature Extraction and Fusion for Emotional Reaction Intensity Estimation and Expression Classification in Videos with Transformers

Jia Li,Yin Chen,Xuesong Zhang,Jiantao Nie,Ziqiang Li,Yangchen Yu,Yan Zhang,Richang Hong,Meng Wang

DOI: https://doi.org/10.48550/arXiv.2303.09164

2023-03-16

Computer Vision and Pattern Recognition

Abstract:In this paper, we present our advanced solutions to the two sub-challenges of Affective Behavior Analysis in the wild (ABAW) 2023: the Emotional Reaction Intensity (ERI) Estimation Challenge and Expression (Expr) Classification Challenge. ABAW 2023 aims to tackle the challenge of affective behavior analysis in natural contexts, with the ultimate goal of creating intelligent machines and robots that possess the ability to comprehend human emotions, feelings, and behaviors. For the Expression Classification Challenge, we propose a streamlined approach that handles the challenges of classification effectively. However, our main contribution lies in our use of diverse models and tools to extract multimodal features such as audio and video cues from the Hume-Reaction dataset. By studying, analyzing, and combining these features, we significantly enhance the model's accuracy for sentiment prediction in a multimodal context. Furthermore, our method achieves outstanding results on the Emotional Reaction Intensity (ERI) Estimation Challenge, surpassing the baseline method by an impressive 84\% increase, as measured by the Pearson Coefficient, on the validation dataset.
Efficient Feature Extraction and Late Fusion Strategy for Audiovisual Emotional Mimicry Intensity Estimation

Jun Yu,Wangyuan Zhu,Jichao Zhu

2024-03-20

Abstract:In this paper, we present the solution to the Emotional Mimicry Intensity (EMI) Estimation challenge, which is part of 6th Affective Behavior Analysis in-the-wild (ABAW) Competition.The EMI Estimation challenge task aims to evaluate the emotional intensity of seed videos by assessing them from a set of predefined emotion categories (i.e., "Admiration", "Amusement", "Determination", "Empathic Pain", "Excitement" and "Joy"). To tackle this challenge, we extracted rich dual-channel visual features based on ResNet18 and AUs for the video modality and effective single-channel features based on Wav2Vec2.0 for the audio modality. This allowed us to obtain comprehensive emotional features for the audiovisual modality. Additionally, leveraging a late fusion strategy, we averaged the predictions of the visual and acoustic models, resulting in a more accurate estimation of audiovisual emotional mimicry intensity. Experimental results validate the effectiveness of our approach, with the average Pearson's correlation Coefficient($\rho$) across the 6 emotion dimensionson the validation set achieving 0.3288.

Multimedia,Machine Learning,Sound,Audio and Speech Processing
Investigation of Multimodal Features, Classifiers and Fusion Methods for Emotion Recognition

Zheng Lian,Ya Li,Jianhua Tao,Jian Huang

DOI: https://doi.org/10.48550/arXiv.1809.06225

2018-09-13

Abstract:Automatic emotion recognition is a challenging task. In this paper, we present our effort for the audio-video based sub-challenge of the Emotion Recognition in the Wild (EmotiW) 2018 challenge, which requires participants to assign a single emotion label to the video clip from the six universal emotions (Anger, Disgust, Fear, Happiness, Sad and Surprise) and Neutral. The proposed multimodal emotion recognition system takes audio, video and text information into account. Except for handcraft features, we also extract bottleneck features from deep neutral networks (DNNs) via transfer learning. Both temporal classifiers and non-temporal classifiers are evaluated to obtain the best unimodal emotion classification result. Then possibilities are extracted and passed into the Beam Search Fusion (BS-Fusion). We test our method in the EmotiW 2018 challenge and we gain promising results. Compared with the baseline system, there is a significant improvement. We achieve 60.34% accuracy on the testing dataset, which is only 1.5% lower than the winner. It shows that our method is very competitive.

Computer Vision and Pattern Recognition,Artificial Intelligence
Multi-modal Expression Recognition with Ensemble Method

Chuanhe Liu,Xinjie Zhang,Xiaolong Liu,Tenggan Zhang,Liyu Meng,Yuchen Liu,Yuanyuan Deng,Wenqiang Jiang

2023-03-17

Abstract:This paper presents our submission to the Expression Classification Challenge of the fifth Affective Behavior Analysis in-the-wild (ABAW) Competition. In our method, multimodal feature combinations extracted by several different pre-trained models are applied to capture more effective emotional information. For these combinations of visual and audio modal features, we utilize two temporal encoders to explore the temporal contextual information in the data. In addition, we employ several ensemble strategies for different experimental settings to obtain the most accurate expression recognition results. Our system achieves the average F1 Score of 0.45774 on the validation set.

Computer Vision and Pattern Recognition
An artificial intelligence-based classifier for musical emotion expression in media education

Jue Lian

DOI: https://doi.org/10.7717/peerj-cs.1472

2023-07-14

PeerJ Computer Science

Abstract:Music can serve as a potent tool for conveying emotions and regulating learners’ moods, while the systematic application of emotional assessment can help to improve teaching efficiency. However, existing music emotion analysis methods based on Artificial Intelligence (AI) rely primarily on pre-marked content, such as lyrics and fail to adequately account for music signals’ perception, transmission, and recognition. To address this limitation, this study first employs sound-level segmentation, data frame processing, and threshold determination to enable intelligent segmentation and recognition of notes. Next, based on the extracted audio features, a Radial Basis Function (RBF) model is utilized to construct a music emotion classifier. Finally, correlation feedback was used to label the classification results further and train the classifier. The study compares the music emotion classification method commonly used in Chinese music education with the Hevner emotion model. It identifies four emotion categories: Quiet, Happy, Sad, and Excited, to classify performers’ emotions. The testing results demonstrate that audio feature recognition time is a mere 0.004 min, with an accuracy rate of over 95%. Furthermore, classifying performers’ emotions based on audio features is consistent with conventional human cognition.

computer science, information systems, artificial intelligence, theory & methods
Facial Expression Recognition Based on Multi-modal Features for Videos in the Wild

Chuanhe Liu,Yuanyuan Deng,Liyu Meng,Wenqiang Jiang,Yuchen Liu,Xinjie Zhang,Tenggan Zhang,Xiaolong Liu

DOI: https://doi.org/10.1109/CVPRW59228.2023.00624

2023-06-01

Abstract:This paper presents our work to the Expression Classification Challenge of the 5th Affective Behavior Analysis in-the-wild (ABAW) Competition. In our method, the multi-modal features are extracted by several different pertained models, which are used to build different combinations to capture more effective emotion information. Specifically, we extracted efficient facial expression features using MAE encoder pre-trained with a large-scale face dataset. For these combinations of visual and audio modal features, we utilize two kinds of temporal encoders to explore the temporal contextual information in the data. In addition, we employ several ensemble strategies for different experimental set-tings to obtain the most accurate expression recognition results. Our system achieves the average F1 Score of 0.4072 on the test set of Aff-wild2 ranking 2nd, which proves the effectiveness of our method.

Computer Science
Affective Behaviour Analysis via Integrating Multi-Modal Knowledge

Wei Zhang,Feng Qiu,Chen Liu,Lincheng Li,Heming Du,Tiancheng Guo,Xin Yu

2024-03-16

Abstract:Affective Behavior Analysis aims to facilitate technology emotionally smart, creating a world where devices can understand and react to our emotions as humans do. To comprehensively evaluate the authenticity and applicability of emotional behavior analysis techniques in natural environments, the 6th competition on Affective Behavior Analysis in-the-wild (ABAW) utilizes the Aff-Wild2, Hume-Vidmimic2, and C-EXPR-DB datasets to set up five competitive tracks, i.e., Valence-Arousal (VA) Estimation, Expression (EXPR) Recognition, Action Unit (AU) Detection, Compound Expression (CE) Recognition, and Emotional Mimicry Intensity (EMI) Estimation. In this paper, we present our method designs for the five tasks. Specifically, our design mainly includes three aspects: 1) Utilizing a transformer-based feature fusion module to fully integrate emotional information provided by audio signals, visual images, and transcripts, offering high-quality expression features for the downstream tasks. 2) To achieve high-quality facial feature representations, we employ Masked-Auto Encoder as the visual features extraction model and fine-tune it with our facial dataset. 3) Considering the complexity of the video collection scenes, we conduct a more detailed dataset division based on scene characteristics and train the classifier for each scene. Extensive experiments demonstrate the superiority of our designs.

Computer Vision and Pattern Recognition
OMG - Emotion Challenge Solution

Yuqi Cui,Xiao Zhang,Yang Wang,Chenfeng Guo,Dongrui Wu

DOI: https://doi.org/10.48550/arXiv.1805.00348

2018-04-30

Abstract:This short paper describes our solution to the 2018 IEEE World Congress on Computational Intelligence One-Minute Gradual-Emotional Behavior Challenge, whose goal was to estimate continuous arousal and valence values from short videos. We designed four base regression models using visual and audio features, and then used a spectral approach to fuse them to obtain improved performance.

Computer Vision and Pattern Recognition,Machine Learning,Sound,Audio and Speech Processing
Emotion Based Image Musicalization

Sicheng Zhao,Hongxun Yao,Fanglin Wang,Xiaolei Jiang,Wei Zhang

DOI: https://doi.org/10.1109/icmew.2014.6890565

2014-01-01

Abstract:Playing appropriate music when watching images can make the images vivid and bring people into their intrinsic world. In this paper, we propose to musicalize images based on their emotions. Most of previous works on image emotion analysis mainly used elements-of-art based low-level visual features, which are vulnerable to the arrangements of elements. Here we propose to extract visual features, inspired by the concept of principles-of-art, to recognize image emotions. To enrich the descriptive power, a dimensional perspective is introduced to emotion modeling. Experiments on the IAPS dataset demonstrate the superiority of the proposed method in comparison to the state-of-the-art methods for emotion regression. The music in MST dataset with approximate emotions to the recognized image emotions is selected to musicalize these images. The user study results show its effectiveness and popularity of the image musicalization method.
Emotion and Theme Recognition in Music with Frequency-Aware RF-Regularized CNNs

Khaled Koutini,Shreyan Chowdhury,Verena Haunschmid,Hamid Eghbal-zadeh,Gerhard Widmer

DOI: https://doi.org/10.48550/arXiv.1911.05833

2019-10-28

Abstract:We present CP-JKU submission to MediaEval 2019; a Receptive Field-(RF)-regularized and Frequency-Aware CNN approach for tagging music with emotion/mood labels. We perform an investigation regarding the impact of the RF of the CNNs on their performance on this dataset. We observe that ResNets with smaller receptive fields -- originally adapted for acoustic scene classification -- also perform well in the emotion tagging task. We improve the performance of such architectures using techniques such as Frequency Awareness and Shake-Shake regularization, which were used in previous work on general acoustic recognition tasks.

Sound,Machine Learning,Multimedia,Audio and Speech Processing
Mutilmodal Feature Extraction and Attention-based Fusion for Emotion Estimation in Videos

Tao Shu,Xinke Wang,Ruotong Wang,Chuang Chen,Yixin Zhang,Xiao Sun

2023-03-18

Abstract:The continuous improvement of human-computer interaction technology makes it possible to compute emotions. In this paper, we introduce our submission to the CVPR 2023 Competition on Affective Behavior Analysis in-the-wild (ABAW). Sentiment analysis in human-computer interaction should, as far as possible Start with multiple dimensions, fill in the single imperfect emotion channel, and finally determine the emotion tendency by fitting multiple results. Therefore, We exploited multimodal features extracted from video of different lengths from the competition dataset, including audio, pose and images. Well-informed emotion representations drive us to propose a Attention-based multimodal framework for emotion estimation. Our system achieves the performance of 0.361 on the validation dataset. The code is available at [<a class="link-external link-https" href="https://github.com/xkwangcn/ABAW-5th-RT-IAI" rel="external noopener nofollow">this https URL</a>].

Computer Vision and Pattern Recognition,Artificial Intelligence
[An improved electroencephalogram feature extraction algorithm and its application in emotion recognition]

Xin Li,Erjuan Cai,Yanxiu Tian,Xiaoqi Sun,Mengdi Fan

DOI: https://doi.org/10.7507/1001-5515.201605066

2017-08-25

Abstract:The result of the emotional state induced by music may provide theoretical support and help for assisted music therapy. The key to assessing the state of emotion is feature extraction of the emotional electroencephalogram (EEG). In this paper, we study the performance optimization of the feature extraction algorithm. A public multimodal database for emotion analysis using physiological signals (DEAP) proposed by Koelstra et al. was applied. Eight kinds of positive and negative emotions were extracted from the dataset, representing the data of fourteen channels from the different regions of brain. Based on wavelet transform, δ, θ, α and β rhythms were extracted. This paper analyzed and compared the performances of three kinds of EEG features for emotion classification, namely wavelet features (wavelet coefficients energy and wavelet entropy), approximate entropy and Hurst exponent. On this basis, an EEG feature fusion algorithm based on principal component analysis (PCA) was proposed. The principal component with a cumulative contribution rate more than 85% was retained, and the parameters which greatly varied in characteristic root were selected. The support vector machine was used to assess the state of emotion. The results showed that the average accuracy rates of emotional classification with wavelet features, approximate entropy and Hurst exponent were respectively 73.15%, 50.00% and 45.54%. By combining these three methods, the features fused with PCA possessed an accuracy of about 85%. The obtained classification accuracy by using the proposed fusion algorithm based on PCA was improved at least 12% than that by using single feature, providing assistance for emotional EEG feature extraction and music therapy.

PKU-AIPL' Solution for MediaEval 2015 Emotion in Music Task.

Beatsens' Solution for MediaEval 2014 Emotion in Music Task.

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Learning Music Emotion Primitives via Supervised Dynamic Clustering.

Multi-Scale Approaches to the MediaEval 2015 "emotion in Music" Task.

Emotion Recognition in Videos via Fusing Multimodal Features.

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

Solution for Emotion Prediction Competition of Workshop on Emotionally and Culturally Intelligent AI

MediaEval 2014: THU-HCSIL Approach to Emotion in Music Task Using Multi-level Regression.

Multimodal Feature Extraction and Fusion for Emotional Reaction Intensity Estimation and Expression Classification in Videos with Transformers

Efficient Feature Extraction and Late Fusion Strategy for Audiovisual Emotional Mimicry Intensity Estimation

Investigation of Multimodal Features, Classifiers and Fusion Methods for Emotion Recognition

Multi-modal Expression Recognition with Ensemble Method

An artificial intelligence-based classifier for musical emotion expression in media education

Facial Expression Recognition Based on Multi-modal Features for Videos in the Wild

Affective Behaviour Analysis via Integrating Multi-Modal Knowledge

OMG - Emotion Challenge Solution

Emotion Based Image Musicalization

Emotion and Theme Recognition in Music with Frequency-Aware RF-Regularized CNNs

Mutilmodal Feature Extraction and Attention-based Fusion for Emotion Estimation in Videos

[An improved electroencephalogram feature extraction algorithm and its application in emotion recognition]