Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

Discovering Gender Differences in Facial Emotion Recognition via Implicit Behavioral Cues

Gender and Emotion Recognition from Implicit User Behavior Signals

Gender and Emotion Recognition with Implicit User Signals

Gender Differences in the Processing of Standard Emotional Visual Stimuli: Integrating ERP and Fmri Results

Exploring Gender Differences in Emotion Recognition with Electroencephalography.

Identifying Gender Differences In Multimodal Emotion Recognition Using Bimodal Deep Autoencoder

Gender Recognition in Emotion Perception Using EEG Features

Exploring neural mechanisms of gender differences in bodily emotion recognition: a time-frequency analysis and network analysis study

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion Recognition

Cross-Subject And Cross-Gender Emotion Classification From Eeg

Action and emotion recognition from point light displays: an investigation of gender differences

Emotion Perception and Recognition: an Exploration of Cultural Differences and Similarities

Emotion Recognition in Conversations Using Brain and Physiological Signals

EEG-Based Brain Computer Interface for Emotion Recognition

Cross-Cultural Emotion Recognition with EEG and Eye Movement Signals Based on Multiple Stacked Broad Learning System

Cultural Differences and Similarities in Emotion Recognition

Awareness is determined by emotion and gender

Gender Differences in Face Recognition:A Behavioral and ERP Study

You're Beautiful When You Smile: Event-Related Brain Potential (ERP) Evidence of Early Opposite-Gender Bias in Happy Faces