Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

Multi-modal Emotion Reaction Intensity Estimation with Temporal Augmentation.

Emotional Reaction Intensity Estimation Based on Multimodal Data

Integrating Holistic and Local Information to Estimate Emotional Reaction Intensity

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

A Dual Branch Network for Emotional Reaction Intensity Estimation

Multimodal Feature Extraction and Fusion for Emotional Reaction Intensity Estimation and Expression Classification in Videos with Transformers

Computer Vision Estimation of Emotion Reaction Intensity in the Wild

Language-guided Multi-modal Emotional Mimicry Intensity Estimation

Mutilmodal Feature Extraction and Attention-based Fusion for Emotion Estimation in Videos

Affective Behaviour Analysis via Integrating Multi-Modal Knowledge

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Exploiting EEG signals and audiovisual feature fusion for video emotion recognition

Multi-modal emotion recognition using EEG and speech signals

An Effective Ensemble Learning Framework for Affective Behaviour Analysis

Efficient Feature Extraction and Late Fusion Strategy for Audiovisual Emotional Mimicry Intensity Estimation

Unimodal Multi-Task Fusion for Emotional Mimicry Intensity Prediction

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

HEU Emotion: A Large-scale Database for Multi-modal Emotion Recognition in the Wild

Facial Expression Recognition Based on Multi-modal Features for Videos in the Wild

Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features

Context-aware Multimodal Fusion for Emotion Recognition