Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

Interpersonal Relationship Analysis with Dyadic EEG Signals via Learning Spatial-Temporal Patterns

Finding neural correlates for social relationships using EEG hyperscanning

DRS-Net: A Spatial–temporal Affective Computing Model Based on Multichannel EEG Data

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

A portable affective computing system for identifying mate preference

Minimizing EEG Human Interference: A Study of an Adaptive EEG Spatial Feature Extraction with Deep Convolutional Neural Networks

A Novel Spatio-Temporal Field for Emotion Recognition Based on EEG Signals

Exploring non-Euclidean relationships in EEG emotion recognition: A dynamic graph attention network

EEG-Based Evaluation of Aesthetic Experience Using BiLSTM Network

An EEG Based Familiar and Unfamiliar Person Identification and Classification System Using Feature Extraction and Directed Functional Brain Network.

SSTD: A Novel Spatio-Temporal Demographic Network for EEG-Based Emotion Recognition

Emotion Recognition from Spatiotemporal EEG Representations with Hybrid Convolutional Recurrent Neural Networks Via Wearable Multi-Channel Headset

A Three-Dimensional Spatio-Temporal Eeg Pattern Analyzing System

EEG Emotion Recognition Based on Ordinary Differential Equation Graph Convolutional Networks and Dynamic Time Wrapping

AITST-Affective EEG-based person identification via interrelated transformer

EEG-based Approach for Recognizing Human Social Emotion Perception

Deep Spatio-Temporal Mutual Learning for EEG Emotion Recognition

Topological EEG Nonlinear Dynamics Analysis for Emotion Recognition

Predicting individual decision-making responses based on single-trial EEG

Multi-Person Brain Activity Recognition via Comprehensive EEG Signal Analysis

Affective EEG-Based Person Identification With Continual Learning