Abstract:This paper presents a novel approach for emotion recognition (ER) based on Electroencephalogram (EEG), Electromyogram (EMG), Electrocardiogram (ECG), and computer vision. The proposed system includes two different models for physiological signals and facial expressions deployed in a real-time embedded system. A custom dataset for EEG, ECG, EMG, and facial expression was collected from 10 participants using an Affective Video Response System. Time, frequency, and wavelet domain-specific features were extracted and optimized, based on their Visualizations from Exploratory Data Analysis (EDA) and Principal Component Analysis (PCA). Local Binary Patterns (LBP), Local Ternary Patterns (LTP), Histogram of Oriented Gradients (HOG), and Gabor descriptors were used for differentiating facial emotions. Classification models, namely decision tree, random forest, and optimized variants thereof, were trained using these features. The optimized Random Forest model achieved an accuracy of 84%, while the optimized Decision Tree achieved 76% for the physiological signal-based model. The facial emotion recognition (FER) model attained an accuracy of 84.6%, 74.3%, 67%, and 64.5% using K-Nearest Neighbors (KNN), Random Forest, Decision Tree, and XGBoost, respectively. Performance metrics, including Area Under Curve (AUC), F1 score, and Receiver Operating Characteristic Curve (ROC), were computed to evaluate the models. The outcome of both results, i.e., the fusion of bio-signals and facial emotion analysis, is given to a voting classifier to get the final emotion. A comprehensive report is generated using the Generative Pretrained Transformer (GPT) language model based on the resultant emotion, achieving an accuracy of 87.5%. The model was implemented and deployed on a Jetson Nano. The results show its relevance to ER. It has applications in enhancing prosthetic systems and other medical fields such as psychological therapy, rehabilitation, assisting individuals with neurological disorders, mental health monitoring, and biometric security.

Edge-enhanced bi-dimensional empirical mode decomposition-based emotion recognition using fusion of feature set

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Electroencephalogram Emotion Recognition Based on Empirical Mode Decomposition and Optimal Feature Selection.

Human facial emotion recognition using improved black hole based extreme learning machine

Multimodal emotion recognition based on the fusion of vision, EEG, ECG, and EMG signals

Decoding Emotions: Integrating EEG Signals and Facial Expressions for Advanced Multimodal Emotion Recognition

Valence-Arousal Model based Emotion Recognition using EEG, peripheral physiological signals and Facial Expression

An Attention-Based Multi-Domain Bi-Hemisphere Discrepancy Feature Fusion Model for EEG Emotion Recognition

Emotion Recognition Using Empirical Mode Decomposition and Approximation Entropy

Multimodal Emotion Recognition from Eye Image, Eye Movement and EEG Using Deep Neural Networks

Speech emotion recognition using multimodal feature fusion with machine learning approach

Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals

Multimodal Emotion Recognition Using Different Fusion Techniques

Multimodal emotion recognition model via hybrid model with improved feature level fusion on facial and EEG feature set

Emotion recognition framework using multiple modalities for an effective human–computer interaction

Optimal Facial Feature Based Emotional Recognition Using Deep Learning Algorithm

Feature-level fusion of multimodal physiological signals for emotion recognition

An efficient facial emotion recognition system using novel deep learning neural network-regression activation classifier

Multimodal Emotion Recognition Based on Feature Selection and Extreme Learning Machine in Video Clips.

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

Emotion Analysis: Bimodal Fusion of Facial Expressions and EEG