Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

TSception: Capturing Temporal Dynamics and Spatial Asymmetry from EEG for Emotion Recognition

TSception: A Deep Learning Framework for Emotion Detection Using EEG

Multi-view Domain-Adaptive Representation Learning for EEG-based Emotion Recognition

EEG Emotion Recognition Based on Self-attention Dynamic Graph Neural Networks

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Hybrid Network Using Dynamic Graph Convolution and Temporal Self-Attention for EEG-Based Emotion Recognition.

Cascaded Convolutional Recurrent Neural Networks for EEG Emotion Recognition Based on Temporal–Frequency–Spatial Features

Multi-Domain Encoding of Spatiotemporal Dynamics in EEG for Emotion Recognition

GTSception: A Deep Learning EEG Emotion Recognition Model Based on Fusion of Global, Time Domain and Frequency Domain Feature Extraction

TSANN-TG: Temporal–Spatial Attention Neural Networks with Task-Specific Graph for EEG Emotion Recognition

Parallel Sequence-Channel Projection Convolutional Neural Network for EEG-Based Emotion Recognition

Spatial-temporal features-based EEG emotion recognition using graph convolution network and long short-term memory

Multi-Channel EEG Based Emotion Recognition Using Temporal Convolutional Network and Broad Learning System

Multi-scale spatiotemporal representation learning for EEG-based emotion recognition

HASTF: a hybrid attention spatio-temporal feature fusion network for EEG emotion recognition

Electroencephalogram-based emotion recognition using factorization temporal separable convolution network

MASA-TCN: Multi-anchor Space-aware Temporal Convolutional Neural Networks for Continuous and Discrete EEG Emotion Recognition

EEG-Based Emotion Recognition Using Spatial-Temporal Graph Convolutional LSTM with Attention Mechanism.

Learning Multiband-Temporal-Spatial EEG Representations of Emotions Using Lightweight Temporal Convolution and 3D Convolutional Neural Network

Spatial-temporal Transformers for EEG Emotion Recognition

EmT: A Novel Transformer for Generalized Cross-subject EEG Emotion Recognition