Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

Decoding the temporal representation of facial expression in face-selective regions

EEG decoding of multidimensional information from emotional faces

Decoding facial expressions based on face-selective and motion-sensitive areas

Neural Responses to Rapid Facial Expressions of Fear and Surprise

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Facial expression recognition: A meta-analytic review of theoretical models and neuroimaging evidence

Decoding the Temporal Structures and Interactions of Multiple Face Dimensions Using Optically Pumped Magnetometer Magnetoencephalography (OPM-MEG)

Layer-specific, retinotopically-diffuse modulation in human visual cortex by emotional faces

Facial Expression Decoding Based on Fmri Brain Signal

Decoding the Temporal Dynamics of Affective Scene Processing

Dissociable Roles of Internal Feelings and Face Recognition Ability in Facial Expression Decoding.

Encoding of facial features by single neurons in the human amygdala and hippocampus

Network Representations of Facial and Bodily Expressions: Evidence from Multivariate Connectivity Pattern Classification.

Functional Integration Of The Posterior Superior Temporal Sulcus Correlates With Facial Expression Recognition

Neural Univariate Activity and Multivariate Pattern in the Posterior Superior Temporal Sulcus Differentially Encode Facial Expression and Identity

Sensory encoding of emotion conveyed by the face and visual context

The Classification of Facial Expressions with Multi-Cultural Backgrounds: an Event-Related Fmri Study

Rapidly Decoding Image Categories from MEG Data Using a Multivariate Short-Time FC Pattern Analysis Approach

Reexamining the neural network involved in perception of facial expression: A meta-analysis

Neural Substrates for Explicit Recognition of Dynamic Facial Expression by Fmri

Decoding six basic emotions from brain functional connectivity patterns