Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

A Temporal Multi-view Fuzzy Classifier for Fusion Identification on Epileptic Brain Network

Epileptic State Classification by Fusing Hand-Crafted and Deep Learning EEG Features

Coherent Pattern in Multi-Layer Brain Networks: Application to Epilepsy Identification

MMF-NNs: Multi-modal Multi-granularity Fusion Neural Networks for brain networks and its application to epilepsy identification

Multi-modal feature fusion with multi-head self-attention for epileptic EEG signals

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

MSTGC: Multi-Channel Spatio-Temporal Graph Convolution Network for Multi-Modal Brain Networks Fusion

Temporal attention fusion network with custom loss function for EEG--fNIRS classification

Classification of hand movements from EEG using a FusionNet based LSTM network

Incomplete multi-modal brain image fusion for epilepsy classification

Electroencephalography (EEG) based epilepsy diagnosis via multiple feature space fusion using shared hidden space-driven multi-view learning

Multimodal Brain Network Jointly Construction and Fusion for Diagnosis of Epilepsy

A Multi-Channel Feature Fusion CNN-Bi-LSTM Epilepsy EEG Classification and Prediction Model Based on Attention Mechanism

Multiview Feature Fusion Representation for Interictal Epileptiform Spikes Detection

Spatio-Temporal Graph Hubness Propagation Model for Dynamic Brain Network Classification

Multimodal Triplet Attention Network for Brain Disease Diagnosis

Time-space-frequency feature Fusion for 3-channel motor imagery classification

Multi feature fusion network for schizophrenia classification and abnormal brain network recognition

Dynamic Functional Connectivity Neural Network for Epileptic Seizure Prediction Using Multi-Channel EEG Signal

Analysis of EEG Signals Fused With CNN and Complex Networks

Fusing Structural and Functional Connectivities using Disentangled VAE for Detecting MCI