Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

Hybrid Emotion-Aware Monitoring System Based on Brainwaves for Internet of Medical Things

IoT-Based Remote Pain Monitoring System: from Device to Cloud Platform.

Emotion-Aware and Intelligent Internet of Medical Things Toward Emotion Recognition During COVID-19 Pandemic

An EEG-Based brain informatics application for enhancing music experience

A Health Monitoring System Based on Flexible Triboelectric Sensors for Intelligence Medical Internet of Things and its Applications in Virtual Reality

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Personalized Emotion Detection using IoT and Machine Learning

Wearable Wireless Dual-Channel EEG System for Emotion Recognition Based on Machine Learning

Affective Computing for eHealth Using Low-Cost Remote Internet of Things-Based EMG Platform

IEMS: an IoT-Empowered Wearable Multimodal Monitoring System in Neurocritical Care

A Multimodal Emotion-Focused e-health Monitoring Support System

Cognimeter: Eeg-Based Brain States Monitoring

An Internet of Medical Things-Based Mental Disorder Prediction System Using EEG Sensor and Big Data Mining

An EEG-Based Brain Computer Interface for Emotion Recognition and Its Application in Patients with Disorder of Consciousness

Multimodal Real-Time patient emotion recognition system using facial expressions and brain EEG signals based on Machine learning and Log-Sync methods

Emotion Recognition Based on Double Tree Complex Wavelet Transform and Machine Learning in Internet of Things

EEG-Based Mental Workload Classification Method Based on Hybrid Deep Learning Model under IoT.

IoT-enabled Low Communication Cost Sentiment Analysis Architecture for Intelligent Healthcare

An adaptive cognitive sensor node for ECG monitoring in the Internet of Medical Things

Automatic Emotion Perception Using Eye Movement Information for E-Healthcare Systems