Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

STM-Net Based Spatial-Temporal Multi-Modal Fusion Network for Emotion Recognition

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Multimodal Fused Emotion Recognition about Expression-EEG Interaction and Collaboration Using Deep Learning

Multimodal Emotion Recognition based on the Fusion of EEG Signals and Eye Movement Data

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

STAFNet: an adaptive multi-feature learning network via spatiotemporal fusion for EEG-based emotion recognition

Emotion Recognition Using Cross-Modal Attention from Eeg and Facial Expression

MF-Net: a multimodal fusion network for emotion recognition based on multiple physiological signals

A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals

STSNet: a Novel Spatio-Temporal-spectral Network for Subject-Independent EEG-based Emotion Recognition

Emotion recognition using spatial-temporal EEG features through convolutional graph attention network

Emotion Analysis: Bimodal Fusion of Facial Expressions and EEG

Multimodal Multi-View Spectral-Spatial-Temporal Masked Autoencoder for Self-Supervised Emotion Recognition

Emotion Recognition Based on Fusion of Long Short-Term Memory Networks and SVMs.

Multi-head attention fusion networks for multi-modal speech emotion recognition

Multi-Modal Emotion Recognition Based On deep Learning Of EEG And Audio Signals

Cross-Modal Guiding Neural Network for Multimodal Emotion Recognition From EEG and Eye Movement Signals

MTLFuseNet: A novel emotion recognition model based on deep latent feature fusion of EEG signals and multi-task learning

Emotion Recognition from Multiple Physiological Signals Using Intra- and Inter-Modality Attention Fusion Network

A Novel DE-CNN-BiLSTM Multi-Fusion Model for EEG Emotion Recognition