Abstract:This paper presents a novel approach for emotion recognition (ER) based on Electroencephalogram (EEG), Electromyogram (EMG), Electrocardiogram (ECG), and computer vision. The proposed system includes two different models for physiological signals and facial expressions deployed in a real-time embedded system. A custom dataset for EEG, ECG, EMG, and facial expression was collected from 10 participants using an Affective Video Response System. Time, frequency, and wavelet domain-specific features were extracted and optimized, based on their Visualizations from Exploratory Data Analysis (EDA) and Principal Component Analysis (PCA). Local Binary Patterns (LBP), Local Ternary Patterns (LTP), Histogram of Oriented Gradients (HOG), and Gabor descriptors were used for differentiating facial emotions. Classification models, namely decision tree, random forest, and optimized variants thereof, were trained using these features. The optimized Random Forest model achieved an accuracy of 84%, while the optimized Decision Tree achieved 76% for the physiological signal-based model. The facial emotion recognition (FER) model attained an accuracy of 84.6%, 74.3%, 67%, and 64.5% using K-Nearest Neighbors (KNN), Random Forest, Decision Tree, and XGBoost, respectively. Performance metrics, including Area Under Curve (AUC), F1 score, and Receiver Operating Characteristic Curve (ROC), were computed to evaluate the models. The outcome of both results, i.e., the fusion of bio-signals and facial emotion analysis, is given to a voting classifier to get the final emotion. A comprehensive report is generated using the Generative Pretrained Transformer (GPT) language model based on the resultant emotion, achieving an accuracy of 87.5%. The model was implemented and deployed on a Jetson Nano. The results show its relevance to ER. It has applications in enhancing prosthetic systems and other medical fields such as psychological therapy, rehabilitation, assisting individuals with neurological disorders, mental health monitoring, and biometric security.

ERIT Lightweight Multimodal Dataset for Elderly Emotion Recognition and Multimodal Fusion Evaluation

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Smile upon the Face but Sadness in the Eyes: Emotion Recognition based on Facial Expressions and Eye Behaviors

EmoWear: Wearable Physiological and Motion Dataset for Emotion Recognition and Context Awareness

Multimodal emotion recognition based on the fusion of vision, EEG, ECG, and EMG signals

Multimodal Adaptive Emotion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels

An Effective Data Fusion Methodology for Multi-modal Emotion Recognition: A Survey

Multi-modal Emotion Reaction Intensity Estimation with Temporal Augmentation.

An Assessment of In-the-Wild Datasets for Multimodal Emotion Recognition

Emotional Reaction Intensity Estimation Based on Multimodal Data

MDEAW: A Multimodal Dataset for Emotion Analysis through EDA and PPG signals from wireless wearable low-cost off-the-shelf Devices

AffectGPT: Dataset and Framework for Explainable Multimodal Emotion Recognition

M-MELD: A Multilingual Multi-Party Dataset for Emotion Recognition in Conversations

Multi-Modal Emotion Recognition Feature Extraction and Data Fusion Methods Evaluation

A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition

HEU Emotion: A Large-scale Database for Multi-modal Emotion Recognition in the Wild

Emotion Recognition from Multimodal Data: a machine learning approach combining classical and hybrid deep architectures

Context-aware Multimodal Fusion for Emotion Recognition

Real-time emotional health detection using fine-tuned transfer networks with multimodal fusion

Multi-modal emotion recognition using EEG and speech signals

A Multimodal Dataset for Mixed Emotion Recognition