Abstract:Modeling correlations between multimodal physiological signals [e.g., canonical correlation analysis (CCA)] for emotion recognition has attracted much attention. However, existing studies rarely consider the neural nature of emotional responses within physiological signals. Furthermore, during fusion space construction, the CCA method maximizes only the correlations between different modalities and neglects the discriminative information of different emotional states. Most importantly, temporal mismatches between different neural activities are often ignored; therefore, the theoretical assumptions that multimodal data should be aligned in time and space before fusion are not fulfilled. To address these issues, we propose a discriminative correlation fusion method coupled with a temporal alignment mechanism for multimodal physiological signals. We first use neural signal analysis techniques to construct neural representations of the central nervous system (CNS) and autonomic nervous system (ANS). respectively. Then, emotion class labels are introduced in CCA to obtain more discriminative fusion representations from multimodal neural responses, and the temporal alignment between the CNS and ANS is jointly optimized with a fusion procedure that applies the Bayesian algorithm. The experimental results demonstrate that our method significantly improves the emotion recognition performance. Additionally, we show that this fusion method can model the underlying mechanisms in human nervous systems during emotional responses, and our results are consistent with prior findings. This study may guide a new approach for exploring human cognitive function based on physiological signals at different time scales and promote the development of computational intelligence and harmonious human-computer interactions.

Multi-view Emotion Recognition Using Deep Canonical Correlation Analysis

Multi-view Domain-Adaptive Representation Learning for EEG-based Emotion Recognition

Multimodal Emotion Recognition Using Deep Canonical Correlation Analysis

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Research on feature fusion for emotion recognition based on discriminative canonical correlation analysis

An Improved Canonical Correlation Analysis for EEG Inter-Band Correlation Extraction.

Multimodal Emotion Recognition Using Deep Generalized Canonical Correlation Analysis with an Attention Mechanism

EEG-Based Emotion Recognition Using Partial Directed Coherence Dense Graph Propagation

An Improved Canonical Correlation Analysis for EEG Inter-Band Correlation Extraction

Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition

Electroencephalogram Emotion Recognition Based on Empirical Mode Decomposition and Optimal Feature Selection.

Multi-View Common Space Learning For Emotion Recognition In The Wild

Multidimensional Feature in Emotion Recognition Based on Multi-Channel EEG Signals.

Correlation-Driven Multi-Modality Graph Decomposition for Cross-Subject Emotion Recognition

Multichannel EEG-Based Emotion Recognition via Group Sparse Canonical Correlation Analysis

Multi-Source Co-adaptation for EEG-Based Emotion Recognition by Mining Correlation Information

A Novel DE-CNN-BiLSTM Multi-Fusion Model for EEG Emotion Recognition

Emotion Recognition From Multimodal Physiological Signals via Discriminative Correlation Fusion With a Temporal Alignment Mechanism

EEG-Based Emotion Recognition Using a 2D CNN with Different Kernels

Multimodal Emotion Recognition from Eye Image, Eye Movement and EEG Using Deep Neural Networks

Emotion Recognition From Arbitrary View Facial Images