Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

An On-Board Executable Multi-Feature Transfer-Enhanced Fusion Model for Three-Lead EEG Sensor-Assisted Depression Diagnosis

An Intra- and Inter-Emotion Transformer-Based Fusion Model with Homogeneous and Diverse Constraints Using Multi-Emotional Audiovisual Features for Depression Detection.

DISTINGUISHING BIPOLAR DEPRESSION FROM MAJOR DEPRESSIVE DISORDER USING FNIRS AND DEEP NEURAL NETWORK

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Depression Detection Based on the Temporal-Spatial-frequency Feature Fusion of EEG

Development of Artificial Intelligence for Determining Major Depressive Disorder Based on Resting-State EEG and Single-Pulse Transcranial Magnetic Stimulation-Evoked EEG Indices

Depression Recognition From EEG Signals Using an Adaptive Channel Fusion Method Via Improved Focal Loss

An Intelligent Depression Detection Model Based On Multimodal Fusion Technology

Multi-Modal Adaptive Fusion Transformer Network for the Estimation of Depression Level

Content-based multiple evidence fusion on EEG and eye movements for mild depression recognition

Mutual Information Based Fusion Model (MIBFM): Mild Depression Recognition Using EEG and Pupil Area Signals

Fusion Based Depression Detection through Artificial Intelligence using Electroencephalogram (EEG)

A novel EEG-based major depressive disorder detection framework with two-stage feature selection

Machine-Learning-Based Depression Detection Model from Electroencephalograph (EEG) Data Obtained by Consumer-Grade EEG Device

Depression Assessment Method: An EEG Emotion Recognition Framework Based on Spatiotemporal Neural Network

EEG-based high-performance depression state recognition

Feature-level fusion approaches based on multimodal EEG data for depression recognition

A machine learning based depression screening framework using temporal domain features of the electroencephalography signals

The Diagnosis of Major Depressive Disorder through Wearable fNIRS by Using Wavelet Transform and Parallel-CNN Feature Fusion

Automatic Diagnosis of Major Depressive Disorder Using a High- and Low-Frequency Feature Fusion Framework

Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data