Updating recommendations for prevention of coronary heart disease.

A. Zanchetti

DOI: https://doi.org/10.1097/00004872-199816100-00001

IF: 4.9

1998-10-01

Journal of Hypertension

Abstract:

What problem does this paper attempt to address?

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Guanghao Yin,Shouqian Sun,Dian Yu,Dejian Li,Kejun Zhang

DOI: https://doi.org/10.1145/3490686

IF: 4.094

2022-01-01

ACM Transactions on Multimedia Computing Communications and Applications

Abstract:Considerable attention has been paid for physiological signal-based emotion recognition in field of affective computing. For the reliability and user friendly acquisition, Electrodermal Activity (EDA) has great advantage in practical applications. However, the EDA-based emotion recognition with hundreds of subjects still lacks effective solution. In this paper, our work makes an attempt to fuse the subject individual EDA features and the external evoked music features. And we propose an end-to-end multimodal framework, the 1-dimensional residual temporal and channel attention network (RTCAN-1D). For EDA features, the novel convex optimization-based EDA (CvxEDA) method is applied to decompose EDA signals into pahsic and tonic signals for mining the dynamic and steady features. The channel-temporal attention mechanism for EDA-based emotion recognition is firstly involved to improve the temporal- and channel-wise representation. For music features, we process the music signal with the open source toolkit openSMILE to obtain external feature vectors. The individual emotion features from EDA signals and external emotion benchmarks from music are fused in the classifing layers. We have conducted systematic comparisons on three multimodal datasets (PMEmo, DEAP, AMIGOS) for 2-classes valance/arousal emotion recognition. Our proposed RTCAN-1D outperforms the existing state-of-the-art models, which also validate that our work provides an reliable and efficient solution for large scale emotion recognition. Our code has been released at this https URL.
Emotion Recognition in Videos via Fusing Multimodal Features.

Shizhe Chen,Yujie Dian,Xinrui Li,Xiaozhu Lin,Qin Jin,Haibo Liu,Li Lu

DOI: https://doi.org/10.1007/978-981-10-3005-5_52

2016-01-01

Abstract:Emotion recognition is a challenging task with a wide range of applications. In this paper, we present our system in the CCPR 2016 multimodal emotion recognition challenge. Multimodal features from acoustic signals, facial expressions and speech contents are extracted to recognize the emotion of the character in the video. Among them the facial CNN feature is the most discriminative feature for emotion recognition. We train SVM and random forest classifiers based on each type of features and utilize early and late fusion to combine the different modality features. To deal with the data unbalance issue, we propose to adapt the probability thresholds for each emotion class. The macro precision of our best multimodal fusion system achieves 50.34% on the testing set, which significantly outperforms the baseline of 30.63 %.
Emotion Recognition From Multimodal Physiological Signals Using a Regularized Deep Fusion of Kernel Machine

Xiaowei Zhang,Jinyong Liu,Jian Shen,Shaojie Li,Kechen Hou,Bin Hu,Jin Gao,Tong Zhang,Bin Hu

DOI: https://doi.org/10.1109/tcyb.2020.2987575

IF: 11.8

2021-09-01

IEEE Transactions on Cybernetics

Abstract:These days, physiological signals have been studied more broadly for emotion recognition to realize emotional intelligence in human–computer interaction. However, due to the complexity of emotions and individual differences in physiological responses, how to design reliable and effective models has become an important issue. In this article, we propose a regularized deep fusion framework for emotion recognition based on multimodal physiological signals. After extracting the effective features from different types of physiological signals, we construct ensemble dense embeddings of multimodal features using kernel matrices, and then utilize a deep network architecture to learn task-specific representations for each kind of physiological signal from these ensemble dense embeddings. Finally, a global fusion layer with a regularization term, which can efficiently explore the correlation and diversity among all of the representations in a synchronous optimization process, is designed to fuse generated representations. Experiments on two benchmark datasets show that this framework can improve the performance of subject-independent emotion recognition compared to single-modal classifiers or other fusion methods. Data visualization also demonstrates that the final fusion representation exhibits higher class-separability power for emotion recognition.

automation & control systems,computer science, cybernetics, artificial intelligence
Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Jiafa Chen,Kaiwei Yu,Fei Wang,Zhengxian Zhou,Yifei Bi,Songlin Zhuang,Dawei Zhang

DOI: https://doi.org/10.3390/electronics13071310

IF: 2.9

2024-04-01

Electronics

Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

engineering, electrical & electronic,computer science, information systems,physics, applied
Feature-level fusion of multimodal physiological signals for emotion recognition

Jing Chen,Bin Hu,Lixin Xu,Philip Moore,Yun Su

DOI: https://doi.org/10.1109/bibm.2015.7359713

2015-11-01

Abstract:Objective: This paper aims to use multimodal physiological signals to automatically recognize human emotions, and a novel multimodal feature fusion approach is proposed. Methods: In the proposed approach, significant multimodal features are selected respectively by two comparative feature selection methods: Fisher Criterion Score and Davies-Bouldin index. Emotion recognition is performed on the valence-arousal emotion space by using hidden Markov models (HMMs) and multimodal feature sets. Four physiological modalities, including electroencephalogram (EEG) from central nervous system and peripheral physiological signals (PERI) from peripheral nervous system as shown in the DEAP database, are employed. Results: We show the best recognition accuracies of 85.63% for arousal and 83.98% for valence. The proposed feature fusion approach is compared with decision-level fusion and non-fusion approaches on the same database; and the comparison demonstrates significant improvements in accuracy obtained by the feature fusion approach. Conclusion: Our work supports the observation that the proposed feature-level fusion approach represents a promising methodology for emotion recognition.
Joint low-rank tensor fusion and cross-modal attention for multimodal physiological signals based emotion recognition

Xin Wan,Yongxiong Wang,Zhe Wang,Yiheng Tang,Benke Liu

DOI: https://doi.org/10.1088/1361-6579/ad5bbc

2024-06-27

Physiological Measurement

Abstract:Objective. Physiological signals based emotion recognition is a prominent research domain in the field of human-computer interaction. Previous studies predominantly focused on unimodal data, giving limited attention to the interplay among multiple modalities. Within the scope of multimodal emotion recognition, integrating the information from diverse modalities and leveraging the complementary information are the two essential issues to obtain the robust representations. Approach. Thus, we propose a intermediate fusion strategy for combining low-rank tensor fusion with the cross-modal attention to enhance the fusion of electroencephalogram (EEG), electrooculogram (EOG), electromyography (EMG), and galvanic skin response (GSR). Firstly, handcrafted features from distinct modalities are individually fed to corresponding feature extractors to obtain latent features. Subsequently, low-rank tensor is fused to integrate the information by the modality interaction representation. Finally, a cross-modal attention module is employed to explore the potential relationships between the distinct latent features and modality interaction representation, and recalibrate the weights of different modalities. And the resultant representation is adopted for emotion recognition. Main Results. Furthermore, to validate the effectiveness of the proposed method, we execute subject-independent experiments within the DEAP dataset. The proposed method has achieved the accuracies of 73.82% and 74.55% for valence and arousal classification. Significance. The results of extensive experiments verify the outstanding performance of the proposed method.

engineering, biomedical,biophysics,physiology
Emotion recognition based on multiple physiological signals

Qi Li,Yunqing Liu,Fei Yan,Qiong Zhang,Cong Liu

DOI: https://doi.org/10.1016/j.bspc.2023.104989

IF: 5.1

2023-05-20

Biomedical Signal Processing and Control

Abstract:Physiological signals can more realistically reflect human emotional states. To overcome the limitations imposed in single-modal emotion recognition, emotion recognition of multimodal physiological signals has received increasingly widespread attention. However, the original fusion models usually ignore the different distributions of multiple signals and how to capture complementary features from multimodal information effectively. This paper proposes an effective classification model for multimodal physiological signals to address the above issues based on modeling the heterogeneity and correlation between multimodal signals. First, differential entropy features are extracted from Electroencephalography (EEG) signals and peripheral physiological signals (PPS) such as Electrocardiographic (ECG) signals, Electromyographic (EMG) signals, and other physiological signals. Then, according to the different distributions and frequency characteristics of the acquired signals, the EEG signal features are made into a three-dimensional feature map and input to the neural network to extract the frequency spatial dimension features. Further temporal features are extracted from the peripheral physiological signals using a long and short-term memory network. Finally, the EEG and peripheral physiological signal features were fused and input to a multimodal long and short-term memory network to extract the association between different modalities and perform classification. The experiments were conducted on the benchmark DEAP dataset, and the results showed that the classification accuracy of the proposed model in this paper was 95.89% and 94.99% in the arousal dimension and the valence dimension, respectively, which were 2.77% and 3.11% higher compared to the unimodal EEG model, respectively. This paper also analyzed the effects of different peripheral physiological signals on emotion recognition.

engineering, biomedical
Research on feature fusion for emotion recognition based on discriminative canonical correlation analysis

Chuqi Liu,Chao Li,Ziping Zhao

DOI: https://doi.org/10.1145/3208788.3208804

2018-04-20

Abstract:With the rapid development of emotion recognition, emotion recognition based on EEG signals and physiological signals has drawn much attention from researchers. However, due to the consistency of multi-source information in emotional expression, emotion recognition based on single modal information is still unsatisfactory. Therefore, we proposed a feature fusion algorithm based on Discriminative Canonical correlation analysis, two modes are dealt with simultaneously, the correlation between the two classes of samples is taken as a similarity measure, introduced the class information of the sample, Fully consider the correlation between similar samples and the correlation between different samples. We use the DEAP database and use the DCCA method to fuse the physiological signals and the EEG signals, which greatly improves the classification effect. The classification of liking dimension is 68.21%, which is about 10% higher than other methods and about 2% higher than the CCA model.
MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

Lei Zhu,Yu Ding,Aiai Huang,Xufei Tan,Jianhai Zhang

DOI: https://doi.org/10.1007/s11760-024-03632-0

IF: 1.583

2024-12-06

Signal Image and Video Processing

Abstract:Currently, research on emotion recognition has shown that multi-modal data fusion has advantages in improving the accuracy and robustness of human emotion recognition, outperforming single-modal methods. Despite the promising results of existing methods, significant challenges remain in effectively fusing data from multiple modalities to achieve superior performance. Firstly, existing works tend to focus on generating a joint representation by fusing multi-modal data, with fewer methods considering the specific characteristics of each modality. Secondly, most methods fail to fully capture the intricate correlations among multiple modalities, often resorting to simplistic combinations of latent features. To address these challenges, we propose a novel fusion network for multi-modal emotion recognition. This network enhances the efficacy of multi-modal fusion while preserving the distinct characteristics of each modality. Specifically, a dual-stream multi-scale feature encoding (MFE) is designed to extract emotional information from both electroencephalogram (EEG) and peripheral physiological signals (PPS) temporal slices. Subsequently, a cross-modal global–local feature fusion module (CGFFM) is proposed to integrate global and local information from multi-modal data and then assign different importance to each modality, which makes the fusion data tend to the more important modalities. Meanwhile, the transformer module is employed to further learn the modality-specific information. Moreover, we introduce the adaptive collaboration block (ACB), which optimally leverages both modality-specific and cross-modality relations for enhanced integration and feature representation. Following extensive experiments on the DEAP and DREAMER multimodal datasets, our model achieves state-of-the-art performance.

engineering, electrical & electronic,imaging science & photographic technology
Multimodal Emotion Recognition Model using Physiological Signals

Zhao Yuxuan,Cao Xinyan,Lin Jinlong,Yu Dunshan,Cao Xixin

DOI: https://doi.org/10.1109/taffc.2021.3093923

2019-01-01

Abstract: As an important field of research in Human-Machine Interactions, emotion recognition based on physiological signals has become research hotspots. Motivated by the outstanding performance of deep learning approaches in recognition tasks, we proposed a Multimodal Emotion Recognition Model that consists of a 3D convolutional neural network model, a 1D convolutional neural network model and a biologically inspired multimodal fusion model which integrates multimodal information on the decision level for emotion recognition. We use this model to classify four emotional regions from the arousal valence plane, i.e., low arousal and low valence (LALV), high arousal and low valence (HALV), low arousal and high valence (LAHV) and high arousal and high valence (HAHV) in the DEAP and AMIGOS dataset. The 3D CNN model and 1D CNN model are used for emotion recognition based on electroencephalogram (EEG) signals and peripheral physiological signals respectively, and get the accuracy of 93.53% and 95.86% with the original EEG signals in these two datasets. Compared with the single-modal recognition, the multimodal fusion model improves the accuracy of emotion recognition by 5% ~ 25%, and the fusion result of EEG signals (decomposed into four frequency bands) and peripheral physiological signals get the accuracy of 95.77%, 97.27% and 91.07%, 99.74% in these two datasets respectively. Integrated EEG signals and peripheral physiological signals, this model could reach the highest accuracy about 99% in both datasets which shows that our proposed method demonstrates certain advantages in solving the emotion recognition tasks.
Hierarchical multimodal-fusion of physiological signals for emotion recognition with scenario adaption and contrastive alignment

Jiehao Tang,Zhuang Ma,Kaiyu Gan,Jianhua Zhang,Zhong Yin

DOI: https://doi.org/10.1016/j.inffus.2023.102129

IF: 18.6

2023-11-16

Information Fusion

Abstract:The lack of complementary affective responses from both the central and peripheral nervous systems could limit the performance of emotion recognition with the single-modal physiological signal. However, when integrating multimodalities, a direct fusion may ignore the heterogeneous nature of multiple feature domains from one modality to another. Besides, there is a risk that the distribution of the multimodal physiological responses may vary across different affective scenarios for stimulating an identical emotional category. The inter-individual variation may also increase due to the superposition of the biometric information from the multimodal features. To tackle these issues, we present a hierarchical multimodal network for robust heterogeneous physiological representations (RHPRNet). First, we applied a spatial-frequency pattern extractor to identify the electroencephalogram (EEG) representations in both the spatial and frequency domains. Next, inter-domain and inter-modality affective encoders are separately applied to the statistic-complexity EEG features and multimodal peripheral features, respectively. All the learned representations are integrated via a hierarchical fusion module. To model the multi-peak patterns stimulated by different affective scenarios, we designed a scenario-adapting pretraining stage. A random contrastive training loss was also applied to mitigate the inter-individual variance. In the end, we performed adequate experiments to examine the performance of the RHPRNet based on three publicly available multimodal databases combined with two validation approaches.

computer science, artificial intelligence, theory & methods
Orofacial Pain and Mastication in Dementia.

F. Lobbezoo,S. Delwel,R. Weijenberg,E. Scherder

DOI: https://doi.org/10.2174/1567205013666160602233535

2017-05-01

Current Alzheimer Research

Abstract:Orofacial pain is a common condition in the general population. It is likely that this is also the case in older persons with a dementia. However, the assessment of (orofacial) pain in non-verbal individuals is hampered by the subjective nature of pain, and their limited communicative abilities. To overcome this drawback, several tools have been developed for the assessment of pain based on observations of pain-specific facial activities, body movements, and vocalizations. Unfortunately, none of the so far developed observational tools have been designed specifically for the assessment of orofacial pain. While the recent psychometric testing of the Orofacial MOBID Pain Scale did not yield reliable outcomes, the subsequently developed Orofacial Pain Scale for Non-Verbal Individuals (OPS-NVI) is currently being evaluated and shows good promise to be reliable and valid. Besides the assessment of orofacial pain, an important application of this instrument will be the investigation of the probable causal association between impaired chewing and cognitive decline, in which orofacial pain plays a mediating role by its negative influence on chewing ability. The identification of this negative influence will urge opinion leaders and policy makers to improve the oral health status in older persons with a dementia. Ultimately, pain-free oral functioning may lead to a higher quality of life and might help stabilizing or improving cognition in this frail and vulnerable patient population.
Multimodal Emotion Recognition Based on Cascaded Multichannel and Hierarchical Fusion

Xia Liu,Zhijing Xu,Kan Huang

DOI: https://doi.org/10.1155/2023/9645611

2023-01-05

Abstract:Humans express their emotions in a variety of ways, which inspires research on multimodal fusion-based emotion recognition that utilizes different modalities to achieve information complementation. However, extracting deep emotional features from different modalities and fusing them remain a challenging task. It is essential to exploit the advantages of different extraction and fusion approaches to capture the emotional information contained within and across modalities. In this paper, we present a novel multimodal emotion recognition framework called multimodal emotion recognition based on cascaded multichannel and hierarchical fusion (CMC-HF), where visual, speech, and text signals are simultaneously utilized as multimodal inputs. First, three cascaded channels based on deep learning technology perform feature extraction for the three modalities separately to enhance deeper information extraction ability within each modality and improve recognition performance. Second, an improved hierarchical fusion module is introduced to promote intermodality interactions of three modalities and further improve recognition and classification accuracy. Finally, to validate the effectiveness of the designed CMC-HF model, some experiments are conducted to evaluate two benchmark datasets, IEMOCAP and CMU-MOSI. The results show that we achieved an almost 2%∼3.2% increase in accuracy of the four classes for the IEMOCAP dataset as well as an improvement of 0.9%∼2.5% in the average class accuracy for the CMU-MOSI dataset when compared to the existing state-of-the-art methods. The ablation experimental results indicate that the cascaded feature extraction method and the hierarchical fusion method make a significant contribution to multimodal emotion recognition, suggesting that the three modalities contain deeper information interactions of both intermodality and intramodality. Hence, the proposed model has better overall performance and achieves higher recognition efficiency and better robustness.
A novel signal channel attention network for multi-modal emotion recognition

Ziang Du,Xia Ye,Pujie Zhao

DOI: https://doi.org/10.3389/fnbot.2024.1442080

IF: 3.493

2024-09-11

Frontiers in Neurorobotics

Abstract:Physiological signal recognition is crucial in emotion recognition, and recent advancements in multi-modal fusion have enabled the integration of various physiological signals for improved recognition tasks. However, current models for emotion recognition with hyper complex multi-modal signals face limitations due to fusion methods and insufficient attention mechanisms, preventing further enhancement in classification performance. To address these challenges, we propose a new model framework named Signal Channel Attention Network (SCA-Net), which comprises three main components: an encoder, an attention fusion module, and a decoder. In the attention fusion module, we developed five types of attention mechanisms inspired by existing research and performed comparative experiments using the public dataset MAHNOB-HCI. All of these experiments demonstrate the effectiveness of the attention module we addressed for our baseline model in improving both accuracy and F1 score metrics. We also conducted ablation experiments within the most effective attention fusion module to verify the benefits of multi-modal fusion. Additionally, we adjusted the training process for different attention fusion modules by employing varying early stopping parameters to prevent model overfitting.

robotics,computer science, artificial intelligence,neurosciences
Automatic Emotion Recognition Using Temporal Multimodal Deep Learning

Bahareh Nakisa,Mohammad Naim Rastgoo,Andry Rakotonirainy,Frederic Maire,Vinod Chandran

DOI: https://doi.org/10.1109/access.2020.3027026

IF: 3.9

2020-01-01

IEEE Access

Abstract:Emotion recognition using miniaturised wearable physiological sensors has emerged as a revolutionary technology in various applications. However, detecting emotions using the fusion of multiple physiological signals remains a complex and challenging task. When fusing physiological signals, it is essential to consider the ability of different fusion approaches to capture the emotional information contained within and across modalities. Moreover, since physiological signals consist of time-series data, it becomes imperative to consider their temporal structures in the fusion process. In this study, we propose a temporal multimodal fusion approach with a deep learning model to capture the non-linear emotional correlation within and across electroencephalography (EEG) and blood volume pulse (BVP) signals and to improve the performance of emotion classification. The performance of the proposed model is evaluated using two different fusion approaches – early fusion and late fusion. Specifically, we use a convolutional neural network (ConvNet) long short-term memory (LSTM) model to fuse the EEG and BVP signals to jointly learn and explore the highly correlated representation of emotions across modalities, after learning each modality with a single deep network. The performance of the temporal multimodal deep learning model is validated on our dataset collected from smart wearable sensors and is also compared with results of recent studies. The experimental results show that the temporal multimodal deep learning models, based on early and late fusion approaches, successfully classified human emotions into one of four quadrants of dimensional emotions with an accuracy of 71.61% and 70.17%, respectively.

computer science, information systems,telecommunications,engineering, electrical & electronic
Investigating Multisensory Integration in Emotion Recognition Through Bio-Inspired Computational Models

Esma Mansouri Benssassi,Juan Ye

DOI: https://doi.org/10.1109/taffc.2021.3106254

IF: 13.99

2023-01-01

IEEE Transactions on Affective Computing

Abstract:Emotion understanding represents a core aspect of human communication. Our social behaviours are closely linked to expressing our emotions and understanding others’ emotional and mental states through social signals. The majority of the existing work proceeds by extracting meaningful features from each modality and applying fusion techniques either at a feature level or decision level. However, these techniques are incapable of translating the constant talk and feedback between different modalities. Such constant talk is particularly important in continuous emotion recognition, where one modality can predict, enhance and complement the other. This article proposes three multisensory integration models, based on different pathways of multisensory integration in the brain; that is, integration by convergence, early cross-modal enhancement, and integration through neural synchrony. The proposed models are designed and implemented using third-generation neural networks, Spiking Neural Networks (SNN). The models are evaluated using widely adopted, third-party datasets and compared to state-of-the-art multimodal fusion techniques, such as early, late and deep learning fusion. Evaluation results show that the three proposed models have achieved comparable results to the state-of-the-art supervised learning techniques. More importantly, this article demonstrates plausible ways to translate constant talk between modalities during the training phase, which also brings advantages in generalisation and robustness to noise.
A multi-stage dynamical fusion network for multimodal emotion recognition

Sihan Chen,Jiajia Tang,Li Zhu,Wanzeng Kong

DOI: https://doi.org/10.1007/s11571-022-09851-w

2022-08-02

Cognitive Neurodynamics

Abstract:In recent years, emotion recognition using physiological signals has become a popular research topic. Physiological signal can reflect the real emotional state for individual which is widely applied to emotion recognition. Multimodal signals provide more discriminative information compared with single modal which arose the interest of related researchers. However, current studies on multimodal emotion recognition normally adopt one-stage fusion method which results in the overlook of cross-modal interaction. To solve this problem, we proposed a multi-stage multimodal dynamical fusion network (MSMDFN). Through the MSMDFN, the joint representation based on cross-modal correlation is obtained. Initially, the latent and essential interactions among various features extracted independently from multiple modalities are explored based on specific manner. Subsequently, the multi-stage fusion network is designed to split the fusion procedure into multi-stages using the correlation observed before. This allows us to exploit much more fine-grained unimodal, bimodal and trimodal intercorrelations. For evaluation, the MSMDFN was verified on multimodal benchmark DEAP. The experiments indicate that our method outperforms the related one-stage multi-modal emotion recognition works.

neurosciences
Emotion Recognition Based on Weighted Fusion Strategy of Multichannel Physiological Signals

Wei Wei,Qingxuan Jia,Yongli Feng,Gang Chen

DOI: https://doi.org/10.1155/2018/5296523

IF: 3.12

2018-07-05

Computational Intelligence and Neuroscience

Abstract:Emotion recognition is an important pattern recognition problem that has inspired researchers for several areas. Various data from humans for emotion recognition have been developed, including visual, audio, and physiological signals data. This paper proposes a decision-level weight fusion strategy for emotion recognition in multichannel physiological signals. Firstly, we selected four kinds of physiological signals, including Electroencephalography (EEG), Electrocardiogram (ECG), Respiration Amplitude (RA), and Galvanic Skin Response (GSR). And various analysis domains have been used in physiological emotion features extraction. Secondly, we adopt feedback strategy for weight definition, according to recognition rate of each emotion of each physiological signal based on Support Vector Machine (SVM) classifier independently. Finally, we introduce weight in decision level by linear fusing weight matrix with classification result of each SVM classifier. The experiments on the MAHNOB-HCI database show the highest accuracy. The results also provide evidence and suggest a way for further developing a more specialized emotion recognition system based on multichannel data using weight fusion strategy.

mathematical & computational biology,neurosciences
Nemoossis, a new genus for the eastern Atlantic long-fin bonefish Pterothrissusbelloci Cadenat 1937 and a redescription of P.gissu Hilgendorf 1877 from the northwestern Pacific

K. Hidaka,Y. Tsukamoto,Y. Iwatsuki

DOI: https://doi.org/10.1007/s10228-016-0536-5

Ichthyological Research

Abstract:
Multimodal Emotion Recognition From EEG Signals and Facial Expressions

Shuai Wang,Jingzi Qu,Yong Zhang,Yidie Zhang

DOI: https://doi.org/10.1109/access.2023.3263670

IF: 3.9

2023-04-08

IEEE Access

Abstract:Emotion recognition has attracted attention in recent years. It is widely used in healthcare, teaching, human-computer interaction, and other fields. Human emotional features are often used to recognize different emotions. Currently, there is more and more research on multimodal emotion recognition based on the fusion of multiple features. This paper proposes a deep learning model for multimodal emotion recognition based on the fusion of electroencephalogram (EEG) signals and facial expressions to achieve an excellent classification effect. First, a pre-trained convolution neural network (CNN) is used to extract the facial features from the facial expressions. Next, the attention mechanism is introduced to extract more critical facial frame features. Then, we apply CNNs to extract spatial features from original EEG signals, which use a local convolution kernel and a global convolution kernel to learn the features of left and right hemispheres channels and all EEG channels. After feature-level fusion, the fusion features of the facial expression features and EEG features are fed into the classifier for emotion recognition. This paper conducted experiments on the DEAP and MAHNOB-HCI datasets to evaluate the performance of the proposed model. The accuracy of valence dimension classification is 96.63%, and arousal dimension classification is 97.15% on the DEAP dataset, while 96.69% and 96.26% on the MAHNOB-HCI dataset. The experimental results show that the proposed model can effectively recognize emotions.

computer science, information systems,telecommunications,engineering, electrical & electronic

Updating recommendations for prevention of coronary heart disease.

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Emotion Recognition in Videos via Fusing Multimodal Features.

Emotion Recognition From Multimodal Physiological Signals Using a Regularized Deep Fusion of Kernel Machine

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Feature-level fusion of multimodal physiological signals for emotion recognition

Joint low-rank tensor fusion and cross-modal attention for multimodal physiological signals based emotion recognition

Emotion recognition based on multiple physiological signals

Research on feature fusion for emotion recognition based on discriminative canonical correlation analysis

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

Multimodal Emotion Recognition Model using Physiological Signals

Hierarchical multimodal-fusion of physiological signals for emotion recognition with scenario adaption and contrastive alignment

Orofacial Pain and Mastication in Dementia.

Multimodal Emotion Recognition Based on Cascaded Multichannel and Hierarchical Fusion

A novel signal channel attention network for multi-modal emotion recognition

Automatic Emotion Recognition Using Temporal Multimodal Deep Learning

Investigating Multisensory Integration in Emotion Recognition Through Bio-Inspired Computational Models

A multi-stage dynamical fusion network for multimodal emotion recognition

Emotion Recognition Based on Weighted Fusion Strategy of Multichannel Physiological Signals

Nemoossis, a new genus for the eastern Atlantic long-fin bonefish Pterothrissusbelloci Cadenat 1937 and a redescription of P.gissu Hilgendorf 1877 from the northwestern Pacific

Multimodal Emotion Recognition From EEG Signals and Facial Expressions