Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

Emotional Contagion System by Perceiving Human Emotion Based on Physiological Signals

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

An Emotion Recognition System Based on Physiological Signals Obtained by Wearable Sensors

A Multimodal Emotional Communication Based Humans-Robots Interaction System

Human Emotion Classification Based on Multiple Physiological Signals by Wearable System.

Emotional States Based 3-D Fuzzy Atmosfield for Casual Communication Between Humans and Robots

Adolescents Environmental Emotion Perception by Integrating EEG and Eye Movements

EmotionSense

Robot Emotion and Performance Regulation Based on HMM

Emotion Recognition Based on Skin Potential Signals with a Portable Wireless Device

AI-enabled emotion-aware robot: The fusion of smart clothing, edge clouds and robotics

An Ear Wearable Device System for Facial Emotion Recognition Disorders

Real-Time Emotion Detection Via E-See

The Effect of Robot Posture and Idle Motion on Spontaneous Emotional Contagion during Robot-Human Interactions

The Effect of a Humanoid Robot’s Emotional Behaviors on Users’ Emotional Responses: Evidence from Pupillometry and Electroencephalography Measures

Performance Robots—Motion Control and Dynamic Emotion Regulation in Public Spaces

Estimating Emotional Intensity from Body Poses for Human-Robot Interaction

EmoSense: Computational Intelligence Driven Emotion Sensing Via Wireless Channel Data

Intelligent Wearable System with Motion and Emotion Recognition Based on Digital Twin Technology

Conveying Emotions to Robots through Touch and Sound

UGotMe: An Embodied System for Affective Human-Robot Interaction