Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

BrainNets: Human Emotion Recognition Using an Internet of Brian Things Platform.

EEG Emotion Recognition Based on Self-attention Dynamic Graph Neural Networks

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

Design and Implementation of an Emotion Analysis System Based on EEG Signals

Wearable Wireless Dual-Channel EEG System for Emotion Recognition Based on Machine Learning

Using electroencephalogram for emotion recognition based on filter-bank long short-term memory networks

Spatial-temporal features-based EEG emotion recognition using graph convolution network and long short-term memory

Spatial-temporal Network for Fine-grained-level Emotion EEG Recognition

Smart 6G Sensor Network Based Human Emotion Analysis by Machine Learning Architectures

An Efficient Machine Learning-Based Emotional Valence Recognition Approach Towards Wearable EEG

E2ENNet: An end-to-end neural network for emotional brain-computer interface

Human Emotions Analysis and Recognition Using EEG Signals in Response to 360$^\circ$ Videos

Feature Pyramid Networks and Long Short-Term Memory for EEG Feature Map-Based Emotion Recognition

Emotion Recognition Using EEG Signals through the Design of a Dry Electrode Based on the Combination of Type 2 Fuzzy Sets and Deep Convolutional Graph Networks

Depression Assessment Method: An EEG Emotion Recognition Framework Based on Spatiotemporal Neural Network

Analysis of brain areas in emotion recognition from EEG signals with deep learning methods

EEG and fNIRS Signal-Based Emotion Identification by Means of Machine Learning Algorithms During Visual Stimuli Exposure

An innovative EEG-based emotion recognition using a single channel-specific feature from the brain rhythm code method

ReliefF-Based EEG Sensor Selection Methods for Emotion Recognition

Online Learning for Wearable EEG-Based Emotion Classification