Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

Context-Aware EEG-Based Perceived Stress Recognition Based on Emotion Transition Paradigm

Neural Responses During Emotion Transitions and Emotion Regulation

Emo-CNN for Perceiving Stress from Audio Signals: A Brain Chemistry Approach

A Transductive Model-based Stress Recognition Method Using Peripheral Physiological Signals

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Real-Time Psychological Stress Detection According to ECG Using Deep Learning

EEG Based Stress Monitoring

Emotion and Stress Recognition Utilizing Galvanic Skin Response and Wearable Technology: A Real-time Approach for Mental Health Care

A novel physiological feature selection method for emotional stress assessment based on emotional state transition

EmoSense: Revealing True Emotions Through Microgestures

Spatial-Frequency Characteristics of EEG Associated With the Mental Stress in Human-Machine Systems

Wearable EEG-Based Brain–Computer Interface for Stress Monitoring

Stress Recognition Using Non-invasive Technology

Leveraging ECG Signals and Social Media for Stress Detection

Personalized acute stress classification from physiological signals with neural processes

Context-Aware Stress Monitoring using Wearable and Mobile Technologies in Everyday Settings

Enhancing Performance and User Engagement in Everyday Stress Monitoring: A Context-Aware Active Reinforcement Learning Approach

Dynamic-Attention-based EEG State Transition Modeling for Emotion Recognition

Depression Assessment Method: An EEG Emotion Recognition Framework Based on Spatiotemporal Neural Network

Stress Emotion Recognition Based on RSP and EMG Signals

The Role of Emotion Regulation and Awareness in Psychosocial Stress: An EEG-Psychometric Correlational Study