Abstract:Emotion recognition remains an intricate task at the crossroads of psychology and artificial intelligence, necessitating real-time, accurate discernment of implicit emotional states. Here, we introduce a pioneering wearable dual-modal device, synergizing functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) to meet this demand. The first-of-its-kind fNIRS-EEG ensemble exploits a temporal convolutional network (TC-ResNet) that takes 24 fNIRS and 16 EEG channels as input for the extraction and recognition of emotional features. Our system has many advantages including its portability, battery efficiency, wireless capabilities, and scalable architecture. It offers a real-time visual interface for the observation of cerebral electrical and hemodynamic changes, tailored for a variety of real-world scenarios. Our approach is a comprehensive emotional detection strategy, with new designs in system architecture and deployment and improvement in signal processing and interpretation. We examine the interplay of emotions and physiological responses to elucidate the cognitive processes of emotion regulation. An extensive evaluation of 30 subjects under four emotion induction protocols demonstrates our bimodal system's excellence in detecting emotions, with an impressive classification accuracy of 99.81% and its ability to reveal the interconnection between fNIRS and EEG signals. Compared with the latest unimodal identification methods, our bimodal approach shows significant accuracy gains of 0.24% for EEG and 8.37% for fNIRS. Moreover, our proposed TC-ResNet-driven temporal convolutional fusion technique outperforms conventional EEG-fNIRS fusion methods, improving the recognition accuracy from 0.7% to 32.98%. This research presents a groundbreaking advancement in affective computing that combines biological engineering and artificial intelligence. Our integrated solution facilitates nuanced and responsive affective intelligence in practical applications, with far-reaching impacts on personalized healthcare, education, and human–computer interaction paradigms.

Memristive Circuit Design of Sequencer Network for Human Emotion Classification

EMSN: an Energy-Efficient Memristive Sequencer Network for Human Emotion Classification in Mental Health Monitoring

Memristor-Based Hierarchical Attention Network for Multimodal Affective Computing in Mental Health Monitoring

Memristive Circuit Design for Personalized Emotion Generation with Memory and Retrieval Functions

A memristor-based associative memory neural network circuit with emotion effect

Design and Implementation of a Flexible Neuromorphic Computing System for Affective Communication Via Memristive Circuits

Memristor-Based Neural Network Circuit of Memory with Emotional Homeostasis

The Design of Memristive Circuit for Affective Multi-Associative Learning

Memristor-Based Circuit Design of PAD Emotional Space and Its Application in Mood Congruity

Memristive Circuit Design of Brain-Like Emotional Learning and Generation

Memristor-Based Affective Associative Memory Circuit with Emotional Transformation

All-optically Controlled Memristor for Dual-Channel Emotional Circuit and Its Application

Memristor-Based Neural Network Circuit of Associative Memory With Overshadowing and Emotion Congruent Effect

Memristive Circuit Design of Emotional Generation and Evolution Based on Skin-Like Sensory Processor.

Memristor-Based Neural Network Circuit of Emotion Congruent Memory With Mental Fatigue and Emotion Inhibition

Memristive Circuit Design of Brain-Inspired Emotional Evolution Based on Theories of Internal Regulation and External Stimulation

Neuromorphic Circuit Implementation of Operant Conditioning Based on Emotion Generation and Modulation

Design and Implementation of an Emotion Analysis System Based on EEG Signals

Memory Circuit Design, Implementation and Analysis Based on Memristor Full-function Pavlov Associative

A Brain-Inspired In-Memory Computing System for Neuronal Communication Via Memristive Circuits.

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System