Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

Instantaneous Estimation of Momentary Affective Responses Using Neurophysiological Signals and a Spatiotemporal Emotional Intensity Regression Network

DRS-Net: A Spatial–temporal Affective Computing Model Based on Multichannel EEG Data

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Using electroencephalogram for emotion recognition based on filter-bank long short-term memory networks

Toward EEG-Based Objective Assessment of Emotion Intensity

Electroencephalogram Emotion Recognition Based on Empirical Mode Decomposition and Optimal Feature Selection.

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

Spatial-temporal Network for Fine-grained-level Emotion EEG Recognition

A Novel Spatio-Temporal Field for Emotion Recognition Based on EEG Signals

Inter-Brain EEG Feature Extraction and Analysis for Continuous Implicit Emotion Tagging During Video Watching

EEG-Based Emotion Estimate Using Shallow Fully Convolutional Neural Network with Boost Training Strategy

Multi-Domain Encoding of Spatiotemporal Dynamics in EEG for Emotion Recognition

Real-time EEG-based Emotion Monitoring Using Stable Features

Enhancing Emotion Recognition Using Region-Specific Electroencephalogram Data and Dynamic Functional Connectivity

Real-Time Eeg-Based User'S Valence Monitoring

Identifying valence and arousal levels via connectivity between EEG channels

Using Support Vector Regression To Estimate Valence Level From Eeg

An Affective EEG Analysis Method Without Feature Engineering

Exploring EEG Microstates for Affective Computing: Decoding Valence and Arousal Experiences During Video Watching

EEG Microstates and Fnirs Metrics Reveal the Spatiotemporal Joint Neural Processing Features of Human Emotions

EEG Emotion Recognition Network Based on Attention and Spatiotemporal Convolution