Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

WiFE: WiFi and Vision based Intelligent Facial-Gesture Emotion Recognition

WiFace: Facial Expression Recognition Using Wi-Fi Signals

WiFE: WiFi and Vision based Unobtrusive Emotion Recognition via Gesture and Facial Expression

WiFi and Vision Enabled Multimodal Emotion Recognition

Bi-modal Emotion Recognition Based on Vision-CSI

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

EmoSense: Computational Intelligence Driven Emotion Sensing Via Wireless Channel Data

Emotion Recognition via Environmental Context and Human Body

Emotion Detection through Body Gesture and Face

Facial Expression Recognition and Positive Emotion Incentive System for Human-Robot Interaction

Bi-modality Fusion for Emotion Recognition in the Wild.

AI in Pursuit of Happiness, Finding Only Sadness: Multi-Modal Facial Emotion Recognition Challenge

Semantic-Rich Facial Emotional Expression Recognition

Facial Emotion Recognition Based on Brain and Machine Collaborative Intelligence

Valence-Arousal Model based Emotion Recognition using EEG, peripheral physiological signals and Facial Expression

EmotioNet Challenge: Recognition of facial expressions of emotion in the wild

Emotion Recognition Using Cross-Modal Attention from Eeg and Facial Expression

Combining Multimodal Features Within A Fusion Network For Emotion Recognition In The Wild

A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference

Facial expression recognition based on deep convolution long short-term memory networks of double-channel weighted mixture

EmoSense: Data-Driven Emotion Sensing Via Off-the-Shelf WiFi Devices.