Nemoossis, a new genus for the eastern Atlantic long-fin bonefish Pterothrissusbelloci Cadenat 1937 and a redescription of P.gissu Hilgendorf 1877 from the northwestern Pacific

K. Hidaka,Y. Tsukamoto,Y. Iwatsuki

DOI: https://doi.org/10.1007/s10228-016-0536-5

Ichthyological Research

Abstract:

What problem does this paper attempt to address?

Emotion Recognition in Videos via Fusing Multimodal Features.

Shizhe Chen,Yujie Dian,Xinrui Li,Xiaozhu Lin,Qin Jin,Haibo Liu,Li Lu

DOI: https://doi.org/10.1007/978-981-10-3005-5_52

2016-01-01

Abstract:Emotion recognition is a challenging task with a wide range of applications. In this paper, we present our system in the CCPR 2016 multimodal emotion recognition challenge. Multimodal features from acoustic signals, facial expressions and speech contents are extracted to recognize the emotion of the character in the video. Among them the facial CNN feature is the most discriminative feature for emotion recognition. We train SVM and random forest classifiers based on each type of features and utilize early and late fusion to combine the different modality features. To deal with the data unbalance issue, we propose to adapt the probability thresholds for each emotion class. The macro precision of our best multimodal fusion system achieves 50.34% on the testing set, which significantly outperforms the baseline of 30.63 %.
MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

Lei Zhu,Yu Ding,Aiai Huang,Xufei Tan,Jianhai Zhang

DOI: https://doi.org/10.1007/s11760-024-03632-0

IF: 1.583

2024-12-06

Signal Image and Video Processing

Abstract:Currently, research on emotion recognition has shown that multi-modal data fusion has advantages in improving the accuracy and robustness of human emotion recognition, outperforming single-modal methods. Despite the promising results of existing methods, significant challenges remain in effectively fusing data from multiple modalities to achieve superior performance. Firstly, existing works tend to focus on generating a joint representation by fusing multi-modal data, with fewer methods considering the specific characteristics of each modality. Secondly, most methods fail to fully capture the intricate correlations among multiple modalities, often resorting to simplistic combinations of latent features. To address these challenges, we propose a novel fusion network for multi-modal emotion recognition. This network enhances the efficacy of multi-modal fusion while preserving the distinct characteristics of each modality. Specifically, a dual-stream multi-scale feature encoding (MFE) is designed to extract emotional information from both electroencephalogram (EEG) and peripheral physiological signals (PPS) temporal slices. Subsequently, a cross-modal global–local feature fusion module (CGFFM) is proposed to integrate global and local information from multi-modal data and then assign different importance to each modality, which makes the fusion data tend to the more important modalities. Meanwhile, the transformer module is employed to further learn the modality-specific information. Moreover, we introduce the adaptive collaboration block (ACB), which optimally leverages both modality-specific and cross-modality relations for enhanced integration and feature representation. Following extensive experiments on the DEAP and DREAMER multimodal datasets, our model achieves state-of-the-art performance.

engineering, electrical & electronic,imaging science & photographic technology
MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

Ziping Zhao,Tian Gao,Haishuai Wang,Björn Schuller

DOI: https://doi.org/10.21437/interspeech.2024-1735

2024-01-01

Abstract:Emotion recognition in conversation should not rely solely on discovering emotion keywords but also make comprehensive judgments after considering the context. To this end, we propose the MFDR to efficiently integrate acoustic and textual information. Specifically, acoustic-word combination and context perception are modeled sequentially in stages through the Sliding Adaptive Window Attention (SAWA) and Gated Context Perception Unit. More importantly, without additional memory overhead, SAWA allows the perception range to be adaptively adjusted according to the correlation strength to solve the misalignment and information loss caused by window truncation, modeling fusion under variable granularity. Furthermore, emotion refinement through Dynamic Frame Convolution strips out emotion-irrelevant frames, thereby generating a compact and emotionally discriminative fusion representation. The efficacy of MFDR is confirmed by IEMOCAP and CMU-MOSEI, where it demonstrates promising performance.
Multimodal Emotion Recognition based on the Fusion of EEG Signals and Eye Movement Data

Meiyi Cheng,Yu Song,Lipeng Feng,Wenyu Zhang,Xiaolin Song

DOI: https://doi.org/10.1109/CCSSTA62096.2024.10691734

2024-07-21

Abstract:Emotion recognition research plays a crucial role in advancing brain-computer interfaces. However, many current methods either rely on a single modal signal or use a multimodal fusion approach that may not fully consider the balance of information between modalities. In this study, we propose a novel multimodal emotion recognition method that combines electroencephalograph (EEG) signals and eye movement data using a cosine loss fusion approach. To extract deep features from the two modalities, we utilize ResNet 18 with a gated attention mechanism for EEG feature extraction, and DenseNet for eye movement data feature extraction. The cosine loss function is employed to address inter-modal differences and ensure a balanced fusion of information. Our proposed method achieves an impressive classification accuracy of $91.16 \%$ for the emotions of happiness, fear, sadness, and neutrality on the SEED-IV dataset. This outperforms both EEG signals (74.48%) and eye movement data $(68.38 \%)$, demonstrating the effectiveness of our approach in emotion recognition tasks.

Computer Science
E-MFNN: an emotion-multimodal fusion neural network framework for emotion recognition

Zhuen Guo,Mingqing Yang,Li Lin,Jisong Li,Shuyue Zhang,Qianbo He,Jiaqi Gao,Heling Meng,Xinran Chen,Yuehao Tao,Chen Yang

DOI: https://doi.org/10.7717/peerj-cs.1977

2024-04-19

PeerJ Computer Science

Abstract:Emotional recognition is a pivotal research domain in computer and cognitive science. Recent advancements have led to various emotion recognition methods, leveraging data from diverse sources like speech, facial expressions, electroencephalogram (EEG), electrocardiogram, and eye tracking (ET). This article introduces a novel emotion recognition framework, primarily targeting the analysis of users’ psychological reactions and stimuli. It is important to note that the stimuli eliciting emotional responses are as critical as the responses themselves. Hence, our approach synergizes stimulus data with physical and physiological signals, pioneering a multimodal method for emotional cognition. Our proposed framework unites stimulus source data with physiological signals, aiming to enhance the accuracy and robustness of emotion recognition through data integration. We initiated an emotional cognition experiment to gather EEG and ET data alongside recording emotional responses. Building on this, we developed the Emotion-Multimodal Fusion Neural Network (E-MFNN), optimized for multimodal data fusion to process both stimulus and physiological data. We conducted extensive comparisons between our framework’s outcomes and those from existing models, also assessing various algorithmic approaches within our framework. This comparison underscores our framework’s efficacy in multimodal emotion recognition. The source code is publicly available at https://figshare.com/s/8833d837871c78542b29 .

computer science, information systems, artificial intelligence, theory & methods
Multi-modal fusion network with complementarity and importance for emotion recognition

Shuai Liu,Peng Gao,Yating Li,Weina Fu,Weiping Ding

DOI: https://doi.org/10.1016/j.ins.2022.11.076

IF: 8.1

2022-11-19

Information Sciences

Abstract:Multimodal emotion recognition, that is, emotion recognition uses machine learning to generate multi-modal features on the basis of videos which has become a research hotspot in the field of artificial intelligence. Traditional multi-modal emotion recognition method only simply connects multiple modalities, and the interactive utilization rate of modal information is low, and it cannot reflect the real emotion under the conflict of modal features well. This article first proves that effective weighting can improve the discrimination between modalities. Therefore, this paper takes into account the importance differences between multiple modalities, and assigns weights to them through the importance attention network. At the same time, considering that there is a certain complementary relationship between the modalities, this paper constructs an attention network with complementary modalities. Finally, the reconstructed features are fused to obtain a multi-modal feature with good interaction. The method proposed in this paper is compared with traditional methods in public datasets. The test results show that our method is accurate in It performs well in both the rate and confusion matrix metrics.

computer science, information systems
Feature-level fusion of multimodal physiological signals for emotion recognition

Jing Chen,Bin Hu,Lixin Xu,Philip Moore,Yun Su

DOI: https://doi.org/10.1109/bibm.2015.7359713

2015-11-01

Abstract:Objective: This paper aims to use multimodal physiological signals to automatically recognize human emotions, and a novel multimodal feature fusion approach is proposed. Methods: In the proposed approach, significant multimodal features are selected respectively by two comparative feature selection methods: Fisher Criterion Score and Davies-Bouldin index. Emotion recognition is performed on the valence-arousal emotion space by using hidden Markov models (HMMs) and multimodal feature sets. Four physiological modalities, including electroencephalogram (EEG) from central nervous system and peripheral physiological signals (PERI) from peripheral nervous system as shown in the DEAP database, are employed. Results: We show the best recognition accuracies of 85.63% for arousal and 83.98% for valence. The proposed feature fusion approach is compared with decision-level fusion and non-fusion approaches on the same database; and the comparison demonstrates significant improvements in accuracy obtained by the feature fusion approach. Conclusion: Our work supports the observation that the proposed feature-level fusion approach represents a promising methodology for emotion recognition.
Multimodal emotion recognition from facial expression and speech based on feature fusion

Guichen Tang,Yue Xie,Ke Li,Ruiyu Liang,Li Zhao

DOI: https://doi.org/10.1007/s11042-022-14185-0

IF: 2.577

2022-11-12

Multimedia Tools and Applications

Abstract:Multimodal emotion recognition is designed to use expression and speech information to identify individual behaviors. Feature fusion can enrich various modal information, which is an important method for multimodal emotion recognition. However, there are several modal information synchronizations and overfitting problems due to large feature dimensions. So, an attention mechanism is introduced to automate the network to pay attention to local effective information. It is used to perform audio and video feature fusion tasks and timing modeling tasks in the network. The main contributions are as follows: 1) the multi-head self-attention mechanism is used for feature fusion of audio and video data to avoid the influence of prior information on the fusion results, and 2) a bidirectional gated recurrent unit is used to model the time series of fusion features; furthermore, the autocorrelation coefficient in the time dimension is also calculated as attention for fusion. Experiment results show that the adopted attention mechanism can effectively improve the accuracy of multimodal emotion recognition.

computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering
Multimodal Emotion Recognition From EEG Signals and Facial Expressions

Shuai Wang,Jingzi Qu,Yong Zhang,Yidie Zhang

DOI: https://doi.org/10.1109/access.2023.3263670

IF: 3.9

2023-04-08

IEEE Access

Abstract:Emotion recognition has attracted attention in recent years. It is widely used in healthcare, teaching, human-computer interaction, and other fields. Human emotional features are often used to recognize different emotions. Currently, there is more and more research on multimodal emotion recognition based on the fusion of multiple features. This paper proposes a deep learning model for multimodal emotion recognition based on the fusion of electroencephalogram (EEG) signals and facial expressions to achieve an excellent classification effect. First, a pre-trained convolution neural network (CNN) is used to extract the facial features from the facial expressions. Next, the attention mechanism is introduced to extract more critical facial frame features. Then, we apply CNNs to extract spatial features from original EEG signals, which use a local convolution kernel and a global convolution kernel to learn the features of left and right hemispheres channels and all EEG channels. After feature-level fusion, the fusion features of the facial expression features and EEG features are fed into the classifier for emotion recognition. This paper conducted experiments on the DEAP and MAHNOB-HCI datasets to evaluate the performance of the proposed model. The accuracy of valence dimension classification is 96.63%, and arousal dimension classification is 97.15% on the DEAP dataset, while 96.69% and 96.26% on the MAHNOB-HCI dataset. The experimental results show that the proposed model can effectively recognize emotions.

computer science, information systems,telecommunications,engineering, electrical & electronic
The role of interleukin-1 alpha in the pathogenesis of periapical bone destruction in a rat model system.

C. Wang,P. Stashenko

DOI: https://doi.org/10.1111/J.1399-302X.1993.TB00543.X

1993-02-01

Oral Microbiology and Immunology

Abstract:To identify the mediators that stimulate periapical bone resorption following infection, a rat model system was used in which active (rapid) and chronic (slow) phases of bone destruction can be distinguished. Extracts of inflammatory tissues from active lesions contained high levels of bone-resorbing activity, which was destroyed by proteinase K and heat (70 degrees C), but was unaffected by polymyxin B, indicating the presence of protein mediator(s) rather than lipopolysaccharide. Fast-performance liquid chromatography gel filtration of extracts of active lesions demonstrated that most activity was associated with macromolecules of MW 30-60 kDa and 15-20 kDa, consistent with bone resorptive cytokines, including interleukin 1 (IL-1) and tumor necrosis factor (TNF). Inhibition with cytokine-specific antisera demonstrated that resorbing activity in active lesions was significantly neutralized by anti-IL-1 alpha, whereas anti-IL-1 beta, anti-TNF alpha and anti-TNF beta had only slight effect. A lower amount of resorbing activity was present in extracts of chronic lesions, which was also neutralized only by anti-IL-1 alpha. Inflammatory tissue explants produced more IL-1 alpha than IL-1 beta in vitro, confirming findings with extracts, and high levels of IL-1 alpha were present in active lesions by radioimmunoassay. These data indicate that bone resorption stimulated by bacterial infection is primarily mediated by IL-1 alpha in this model. The similarity of cytokines in active and chronic lesions suggests that quantitative rather than qualitative differences in these mediators may account for lesion progression.
Multimodal emotion recognition model via hybrid model with improved feature level fusion on facial and EEG feature set

Tripathi, Mukesh Kumar

DOI: https://doi.org/10.1007/s11042-024-19171-2

IF: 2.577

2024-04-27

Multimedia Tools and Applications

Abstract:In recent years, academics have placed a high value on multi-modal emotion identification, as well as extensive research has been conducted in the areas of video, text, voice, and physical signal emotion detection. This paper proposes a novel multimodal emotion recognition model that employs a hybrid model with AMIG-based feature fusion on facial and EEG feature sets. The EEG signal and facial image are subjected to preprocessing to remove unwanted background noises with the Butterworth filter and Viola Jones Algorithm, respectively. While considering the pre-processed EEG signal, features such as EWFS-transform, wavelet features, and CSP-based features are extracted. In the proposed EWFS-transform, the window function is modified by using the frequency function and updated STFT. Conversely, the features including SE-AMM-EST-based features; LGXP and GLCM are extracted while considering the preprocessed face image. In the proposed SE-AMM-EST-based features, the mean of shape is updated and covariance in PCA for extracting texture-based features. In order to extract redundant free essential features, AMIG-based feature fusion is proposed. Further, the fused features are fed into a proposed hybrid model that includes LSTM and AML-CNN models.

computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering
Information Fusion in Attention Networks Using Adaptive and Multi-level Factorized Bilinear Pooling for Audio-visual Emotion Recognition

Hengshun Zhou,Jun Du,Yuanyuan Zhang,Qing Wang,Qing-Feng Liu,Chin-Hui Lee

DOI: https://doi.org/10.48550/arXiv.2111.08910

2021-11-17

Abstract:Multimodal emotion recognition is a challenging task in emotion computing as it is quite difficult to extract discriminative features to identify the subtle differences in human emotions with abstract concept and multiple expressions. Moreover, how to fully utilize both audio and visual information is still an open problem. In this paper, we propose a novel multimodal fusion attention network for audio-visual emotion recognition based on adaptive and multi-level factorized bilinear pooling (FBP). First, for the audio stream, a fully convolutional network (FCN) equipped with 1-D attention mechanism and local response normalization is designed for speech emotion recognition. Next, a global FBP (G-FBP) approach is presented to perform audio-visual information fusion by integrating selfattention based video stream with the proposed audio stream. To improve G-FBP, an adaptive strategy (AG-FBP) to dynamically calculate the fusion weight of two modalities is devised based on the emotion-related representation vectors from the attention mechanism of respective modalities. Finally, to fully utilize the local emotion information, adaptive and multi-level FBP (AMFBP) is introduced by combining both global-trunk and intratrunk data in one recording on top of AG-FBP. Tested on the IEMOCAP corpus for speech emotion recognition with only audio stream, the new FCN method outperforms the state-ofthe-art results with an accuracy of 71.40%. Moreover, validated on the AFEW database of EmotiW2019 sub-challenge and the IEMOCAP corpus for audio-visual emotion recognition, the proposed AM-FBP approach achieves the best accuracy of 63.09% and 75.49% respectively on the test set.

Sound,Multimedia,Audio and Speech Processing
A multi-stage dynamical fusion network for multimodal emotion recognition

Sihan Chen,Jiajia Tang,Li Zhu,Wanzeng Kong

DOI: https://doi.org/10.1007/s11571-022-09851-w

2022-08-02

Cognitive Neurodynamics

Abstract:In recent years, emotion recognition using physiological signals has become a popular research topic. Physiological signal can reflect the real emotional state for individual which is widely applied to emotion recognition. Multimodal signals provide more discriminative information compared with single modal which arose the interest of related researchers. However, current studies on multimodal emotion recognition normally adopt one-stage fusion method which results in the overlook of cross-modal interaction. To solve this problem, we proposed a multi-stage multimodal dynamical fusion network (MSMDFN). Through the MSMDFN, the joint representation based on cross-modal correlation is obtained. Initially, the latent and essential interactions among various features extracted independently from multiple modalities are explored based on specific manner. Subsequently, the multi-stage fusion network is designed to split the fusion procedure into multi-stages using the correlation observed before. This allows us to exploit much more fine-grained unimodal, bimodal and trimodal intercorrelations. For evaluation, the MSMDFN was verified on multimodal benchmark DEAP. The experiments indicate that our method outperforms the related one-stage multi-modal emotion recognition works.

neurosciences
EEG-based Emotion Recognition with Feature Fusion Networks.

Gao Qiang,Yang Yi,Kang Qiaoju,Tian Zekun,Song Yu

DOI: https://doi.org/10.1007/s13042-021-01414-5

2021-01-01

International Journal of Machine Learning and Cybernetics

Abstract:With the rapid development of Human-computer interaction, automatic emotion recognition based on multichannel electroencephalography (EEG) signals has attracted much attention in recent years. However, many existing studies on EEG-based emotion recognition ignore the correlation information between different EEG channels and cannot fully capture the contextual information of EEG signals. In this paper, a novel multi-feature fusion network is proposed, which consists of spatial and temporal neural network structures to learn discriminative spatio-temporal emotional information to recognize emotion. In this experiment, two common types of features, time-domain features (Hjorth, Differential Entropy, Sample Entropy) and frequency domain features (Power Spectral Density), are extracted. Then, to learn the spatial and contextual information, a convolution neural network, inspired by GoogleNet with inception structure, was adopted to capture the intrinsic spatial relationship of EEG electrodes and contextual information, respectively. Fully connected layers are used for feature fusion, instead of the SoftMax function, SVM is selected to classify the high-level emotion features. Finally, to evaluate the proposed method, we conduct leave-one-subject-out EEG emotion recognition experiments on the DEAP dataset, and the experiment results show that the proposed method achieves excellent performance and average emotion recognition accuracies of 80.52% and 75.22% in the valence and arousal classification tasks of the DEAP database, respectively.
MTLFuseNet: A novel emotion recognition model based on deep latent feature fusion of EEG signals and multi-task learning

Rui Li,Chao Ren,Yiqing Ge,Qiqi Zhao,Yikun Yang,Yuhan Shi,Xiaowei Zhang,Bin Hu

DOI: https://doi.org/10.1016/j.knosys.2023.110756

IF: 8.139

2023-07-02

Knowledge-Based Systems

Abstract:How to extract discriminative latent feature representations from electroencephalography (EEG) signals and build a generalized model is a topic in EEG-based emotion recognition research. This study proposed a novel emotion recognition model based on deep latent feature fusion of EEG signals and multi-task learning, referred to as MTLFuseNet. MTLFuseNet learned spatio-temporal latent features of EEG in an unsupervised manner by a variational autoencoder (VAE) and learned the spatio-spectral features of EEG in a supervised manner by a graph convolutional network (GCN) and gated recurrent unit (GRU) network. Afterward, the two latent features were fused to form more complementary and discriminative spatio-temporal-spectral fusion features for EEG signal representation. In addition, MTLFuseNet was constructed based on multi-task learning. The focal loss was introduced to solve the problem of unbalanced sample classes in an emotional dataset, and the triplet-center loss was introduced to make the fused latent feature vectors more discriminative. Finally, a subject-independent leave-one-subject-out cross-validation strategy was used to validate extensively on two public datasets, DEAP and DREAMER. On the DEAP dataset, the average accuracies of valence and arousal are 71.33% and 73.28%, respectively. On the DREAMER dataset, the average accuracies of valence and arousal are 80.43% and 83.33%, respectively. The experimental results show that the proposed MTLFuseNet model achieves excellent recognition performance, outperforming the state-of-the-art methods.

computer science, artificial intelligence
Multi-head attention fusion networks for multi-modal speech emotion recognition

Junfeng Zhang,Lining Xing,Zhen Tan,Hongsen Wang,Kesheng Wang

DOI: https://doi.org/10.1016/j.cie.2022.108078

2022-06-01

Abstract:Multi-modal speech emotion recognition is a study to predict emotion categories by combining speech data with other types of data, such as video, speech text transcription, body action, or facial expression when speaking, which will involve the fusion of multiple features. Most of the early studies, however, directly spliced multi-modal features in the fusion layer after single-modal modeling, resulting in ignoring the connection between speech and other modal features. As a result, we propose a novel multi-modal speech emotion recognition model based on multi-head attention fusion networks, which employs transcribed text and motion capture (MoCap) data involving facial expression, head rotation, and hand action to supplement speech data and perform emotion recognition. In unimodal, we use a two-layer Transformer's encoder combination model to extract speech and text features separately, and MoCap is modeled using a deep residual shrinkage network. Simultaneously, We innovated by changing the input of the Transformer encoder to learn the similarities between speech and text, speech and MoCap, and then output text and MoCap features that are more similar to speech features, and finally, predict the emotion category using combined features. In the IEMOCAP dataset, our model outperformed earlier research with a recognition accuracy of 75.6%.

computer science, interdisciplinary applications,engineering, industrial
HASTF: a hybrid attention spatio-temporal feature fusion network for EEG emotion recognition

Fangzhou Hu,Fei Wang,Jinying Bi,Zida An,Chao Chen,Gangguo Qu,Shuai Han

DOI: https://doi.org/10.3389/fnins.2024.1479570

IF: 4.3

2024-10-15

Frontiers in Neuroscience

Abstract:Introduction: EEG-based emotion recognition has gradually become a new research direction, known as affective Brain-Computer Interface (aBCI), which has huge application potential in human-computer interaction and neuroscience. However, how to extract spatio-temporal fusion features from complex EEG signals and build learning method with high recognition accuracy and strong interpretability is still challenging. Methods: In this paper, we propose a hybrid attention spatio-temporal feature fusion network for EEG-based emotion recognition. First, we designed a spatial attention feature extractor capable of merging shallow and deep features to extract spatial information and adaptively select crucial features under different emotional states. Then, the temporal feature extractor based on the multi-head attention mechanism is integrated to perform spatio-temporal feature fusion to achieve emotion recognition. Finally, we visualize the extracted spatial attention features using feature maps, further analyzing key channels corresponding to different emotions and subjects. Results: Our method outperforms the current state-of-the-art methods on two public datasets, SEED and DEAP. The recognition accuracy are 99.12% ± 1.25% (SEED), 98.93% ± 1.45% (DEAP-arousal), and 98.57% ± 2.60% (DEAP-valence). We also conduct ablation experiments, using statistical methods to analyze the impact of each module on the final result. The spatial attention features reveal that emotion-related neural patterns indeed exist, which is consistent with conclusions in the field of neurology. Discussion: The experimental results show that our method can effectively extract and fuse spatial and temporal information. It has excellent recognition performance, and also possesses strong robustness, performing stably across different datasets and experimental environments for emotion recognition.

neurosciences
Joint low-rank tensor fusion and cross-modal attention for multimodal physiological signals based emotion recognition

Xin Wan,Yongxiong Wang,Zhe Wang,Yiheng Tang,Benke Liu

DOI: https://doi.org/10.1088/1361-6579/ad5bbc

2024-06-27

Physiological Measurement

Abstract:Objective. Physiological signals based emotion recognition is a prominent research domain in the field of human-computer interaction. Previous studies predominantly focused on unimodal data, giving limited attention to the interplay among multiple modalities. Within the scope of multimodal emotion recognition, integrating the information from diverse modalities and leveraging the complementary information are the two essential issues to obtain the robust representations. Approach. Thus, we propose a intermediate fusion strategy for combining low-rank tensor fusion with the cross-modal attention to enhance the fusion of electroencephalogram (EEG), electrooculogram (EOG), electromyography (EMG), and galvanic skin response (GSR). Firstly, handcrafted features from distinct modalities are individually fed to corresponding feature extractors to obtain latent features. Subsequently, low-rank tensor is fused to integrate the information by the modality interaction representation. Finally, a cross-modal attention module is employed to explore the potential relationships between the distinct latent features and modality interaction representation, and recalibrate the weights of different modalities. And the resultant representation is adopted for emotion recognition. Main Results. Furthermore, to validate the effectiveness of the proposed method, we execute subject-independent experiments within the DEAP dataset. The proposed method has achieved the accuracies of 73.82% and 74.55% for valence and arousal classification. Significance. The results of extensive experiments verify the outstanding performance of the proposed method.

engineering, biomedical,biophysics,physiology
Attention-based 3D convolutional recurrent neural network model for multimodal emotion recognition

Yiming Du,Penghai Li,Longlong Cheng,Xuanwei Zhang,Mingji Li,Fengzhou Li

DOI: https://doi.org/10.3389/fnins.2023.1330077

IF: 4.3

2024-01-11

Frontiers in Neuroscience

Abstract:Introduction: Multimodal emotion recognition has become a hot topic in human-computer interaction and intelligent healthcare fields. However, combining information from different human different modalities for emotion computation is still challenging. Methods: In this paper, we propose a three-dimensional convolutional recurrent neural network model (referred to as 3FACRNN network) based on multimodal fusion and attention mechanism. The 3FACRNN network model consists of a visual network and an EEG network. The visual network is composed of a cascaded convolutional neural network–time convolutional network (CNN-TCN). In the EEG network, the 3D feature building module was added to integrate band information, spatial information and temporal information of the EEG signal, and the band attention and self-attention modules were added to the convolutional recurrent neural network (CRNN). The former explores the effect of different frequency bands on network recognition performance, while the latter is to obtain the intrinsic similarity of different EEG samples. Results: To investigate the effect of different frequency bands on the experiment, we obtained the average attention mask for all subjects in different frequency bands. The distribution of the attention masks across the different frequency bands suggests that signals more relevant to human emotions may be active in the high frequency bands γ (31–50 Hz). Finally, we try to use the multi-task loss function Lc to force the approximation of the intermediate feature vectors of the visual and EEG modalities, with the aim of using the knowledge of the visual modalities to improve the performance of the EEG network model. The mean recognition accuracy and standard deviation of the proposed method on the two multimodal sentiment datasets DEAP and MAHNOB-HCI (arousal, valence) were 96.75 ± 1.75, 96.86 ± 1.33; 97.55 ± 1.51, 98.37 ± 1.07, better than those of the state-of-the-art multimodal recognition approaches. Discussion: The experimental results show that starting from the multimodal information, the facial video frames and electroencephalogram (EEG) signals of the subjects are used as inputs to the emotion recognition network, which can enhance the stability of the emotion network and improve the recognition accuracy of the emotion network. In addition, in future work, we will try to utilize sparse matrix methods and deep convolutional networks to improve the performance of multimodal emotion networks.

neurosciences
Hierarchical fusion of visual and physiological signals for emotion recognition

Yuchun Fang,Ruru Rong,Jun Huang

DOI: https://doi.org/10.1007/s11045-021-00774-z

IF: 2.03

2021-04-05

Multidimensional Systems and Signal Processing

Abstract:Emotion recognition is an attractive and essential topic in image and signal processing. In this paper, we propose a multi-level fusion method to combine visual information and physiological signals for emotion recognition. For visual information, we propose a serial fusion of two-stage features to enhance the representation of facial expression in a video sequence. We propose to integrate the Neural Aggregation Network with Convolutional Neural Network feature map to reinforce the vital emotional frames. For physiological signals, we propose a parallel fusion scheme to widen the band of the annotation of the electroencephalogram signals. We extract the frequency feature with the Linear-Frequency Cepstral Coefficients and enhance it with the signal complexity denoted by Sample Entropy (SampEn). In the classification stage, we realize both feature level and decision level fusion of both visual and physiological information. Experimental results validate the effectiveness of the proposed multi-level multi-modal feature representation method.

engineering, electrical & electronic,computer science, theory & methods

Nemoossis, a new genus for the eastern Atlantic long-fin bonefish Pterothrissusbelloci Cadenat 1937 and a redescription of P.gissu Hilgendorf 1877 from the northwestern Pacific

Emotion Recognition in Videos via Fusing Multimodal Features.

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

Multimodal Emotion Recognition based on the Fusion of EEG Signals and Eye Movement Data

E-MFNN: an emotion-multimodal fusion neural network framework for emotion recognition

Multi-modal fusion network with complementarity and importance for emotion recognition

Feature-level fusion of multimodal physiological signals for emotion recognition

Multimodal emotion recognition from facial expression and speech based on feature fusion

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

The role of interleukin-1 alpha in the pathogenesis of periapical bone destruction in a rat model system.

Multimodal emotion recognition model via hybrid model with improved feature level fusion on facial and EEG feature set

Information Fusion in Attention Networks Using Adaptive and Multi-level Factorized Bilinear Pooling for Audio-visual Emotion Recognition

A multi-stage dynamical fusion network for multimodal emotion recognition

EEG-based Emotion Recognition with Feature Fusion Networks.

MTLFuseNet: A novel emotion recognition model based on deep latent feature fusion of EEG signals and multi-task learning

Multi-head attention fusion networks for multi-modal speech emotion recognition

HASTF: a hybrid attention spatio-temporal feature fusion network for EEG emotion recognition

Joint low-rank tensor fusion and cross-modal attention for multimodal physiological signals based emotion recognition

Attention-based 3D convolutional recurrent neural network model for multimodal emotion recognition

Hierarchical fusion of visual and physiological signals for emotion recognition