Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

PR-PL: A Novel Prototypical Representation Based Pairwise Learning Framework for Emotion Recognition Using EEG Signals

Multi-view Domain-Adaptive Representation Learning for EEG-based Emotion Recognition

Electroencephalogram Emotion Recognition Based on Empirical Mode Decomposition and Optimal Feature Selection.

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Subject-Independent EEG Emotion Recognition Based on Genetically Optimized Projection Dictionary Pair Learning

Partial Label Learning for Emotion Recognition from EEG

A novel ensemble learning method using multiple objective particle swarm optimization for subject-independent EEG-based emotion recognition

PARSE: Pairwise Alignment of Representations in Semi-Supervised EEG Learning for Emotion Recognition

EEGMatch: Learning with Incomplete Labels for Semi-Supervised EEG-based Cross-Subject Emotion Recognition

MPED: A Multi-Modal Physiological Emotion Database for Discrete Emotion Recognition

Design and Implementation of an Emotion Analysis System Based on EEG Signals

A Novel Spatio-Temporal Field for Emotion Recognition Based on EEG Signals

Self-Supervised EEG Representation Learning for Robust Emotion Recognition

A Novel Deep-Learning Based Framework for Multi-Subject Emotion Recognition

Expression-Guided EEG Representation Learning for Emotion Recognition

Multi-modal emotion recognition using EEG and speech signals

Emotion recognition with residual network driven by spatial-frequency characteristics of EEG recorded from hearing-impaired adults in response to video clips

GC-STCL: A Granger Causality-Based Spatial–Temporal Contrastive Learning Framework for EEG Emotion Recognition

Possibilistic Clustering-Promoting Semi-Supervised Learning for EEG-Based Emotion Recognition

Emotion Recognition from EEG Signals of Hearing-Impaired People Using Stacking Ensemble Learning Framework Based on a Novel Brain Network

Feature Pyramid Networks and Long Short-Term Memory for EEG Feature Map-Based Emotion Recognition