Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

Intelligent Polygraph Based on Fusion of Electrocardiogram and Electroencephalogram Sensors

ECG Authentication Method Based on Parallel Multi-Scale One-Dimensional Residual Network with Center and Margin Loss.

Lie Detection Using Brain Imaging Technology and Its Psychophysiological Basis

Childhood epilepsy syndromes classification based on fused features of electroencephalogram and electrocardiogram.

Intelligent electrode design for long-term ECG monitoring at home: Prototype design using FPAA and FPGA

Research on Polygraph Technology Based on Ballistocardiogram Signal

Brain Fingerprinting and Lie Detection: A Study of Dynamic Functional Connectivity Patterns of Deception Using EEG Phase Synchrony Analysis

An Automatic Lie Detection Model Using EEG Signals Based on the Combination of Type 2 Fuzzy Sets and Deep Graph Convolutional Networks

ECG Signal Reconstruction Based on Facial Videos Via Combined Explicit and Implicit Supervision.

TrueWatch: Polygraph Examination With Smartwatches

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

An Automatic Cardiac Arrhythmia Classification System with Wearable Electrocardiogram

Deep learning feature representation for electrocardiogram identification

An ECG Signal Acquisition and Analysis System Based on Machine Learning with Model Fusion

A Novel Electrocardiogram Biometric Identification Method Based on Temporal-Frequency Autoencoding

Hybrid Score- and Rank-level Fusion for Person Identification using Face and ECG Data

Modeling and analysis of fatigue detection with multi-channel data fusion

Electrocardiogram-Based Biometric Identification Using Mixed Feature Extraction and Sparse Representation

Prediction of Brain States of Concentration and Relaxation in Real Time with Portable Electroencephalographs

Heart biometrics based on ECG signal by sparse coding and bidirectional long short-term memory

A Graph-Based Feature Extraction Algorithm Towards a Robust Data Fusion Framework for Brain-Computer Interfaces