Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

A Frown-Based Thermal Comfort Detection Method of Facial Emotion Recognition

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Face emotion recognition based on infrared thermal imagery by applying machine learning and parallelism

Efficient Facial Expression Recognition with Representation Reinforcement Network and Transfer Self-Training for Human–Machine Interaction

A Noncontact Emotion Recognition Method Based on Complexion and Heart Rate

Facial expression recognition using adapted residual based deep neural network

Region Based Robust Facial Expression Analysis

Human Emotion Identification Based on Facial Expression using Image Processing

A deep learning model for classifying human facial expressions from infrared thermal images

Spontaneous Facial Expression Recognition by Using Feature-Level Fusion of Visible and Thermal Infrared Images

A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference

A study on computer vision for facial emotion recognition

A method for recognizing facial expression intensity based on facial muscle variations

Mobilenetv3: a deep learning technique for human face expressions identification

Automatic Facial Expression Recognition Based on a Deep Convolutional-Neural-network Structure

Expression Recognition Based on Human Emotion-Attention Circuit

The application of deep learning in computer vision: Facial emotion recognition based on convolutional neural network

Facial Expression Recognition from Infrared Thermal Videos.

Infrared thermal imaging face expression recognition based on Harris algorithm

A Comparison of Methods of Facial Expression Recognition

Human emotion and cognition recognition from body language of the head using soft computing techniques