Abstract:Most human emotion recognition methods largely depend on classifying stereotypical facial expressions that represent emotions. However, such facial expressions do not necessarily correspond to actual emotional states and may correspond to communicative intentions. In other cases, emotions are hidden, cannot be expressed, or may have lower arousal manifested by less pronounced facial expressions, as may occur during passive video viewing. This study improves an emotion classification approach developed in a previous study, which classifies emotions remotely without relying on stereotypical facial expressions or contact-based methods, using short facial video data. In this approach, we desire to remotely sense transdermal cardiovascular spatiotemporal facial patterns associated with different emotional states and analyze this data via machine learning. In this paper, we propose several improvements, which include a better remote heart rate estimation via a preliminary skin segmentation, improvement of the heartbeat peaks and troughs detection process, and obtaining a better emotion classification accuracy by employing an appropriate deep learning classifier using an RGB camera input only with data. We used the dataset obtained in the previous study, which contains facial videos of 110 participants who passively viewed 150 short videos that elicited the following five emotion types: amusement, disgust, fear, sexual arousal, and no emotion, while three cameras with different wavelength sensitivities (visible spectrum, near-infrared, and longwave infrared) recorded them simultaneously. From the short facial videos, we extracted unique high-resolution spatiotemporal, physiologically affected features and examined them as input features with different deep-learning approaches. An EfficientNet-B0 model type was able to classify participants' emotional states with an overall average accuracy of 47.36% using a single input spatiotemporal feature map obtained from a regular RGB camera.

Decoding emotions: unveiling the potential of facial landmarks

Decoding facial expressions based on face-selective and motion-sensitive areas

Observer-generated maps of diagnostic facial features enable categorization and prediction of emotion expressions

Decoding the temporal representation of facial expression in face-selective regions

Finding Emotions in Faces: A Meta-Classifier

Your Face Mirrors Your Deepest Beliefs-Predicting Personality and Morals through Facial Emotion Recognition

Recognizing Emotions with EmotionalDAN

Classification of Individual’s discrete emotions reflected in facial microexpressions using electroencephalogram and facial electromyogram

Decoding Functional Brain Data for Emotion Recognition: A Machine Learning Approach

Synthetic vs Human Emotional Faces: What Changes in Humans' Decoding Accuracy

Face Emotion Recognization Using Dataset Augmentation Based on Neural Network

Decoding six basic emotions from brain functional connectivity patterns

Classifying and Visualizing Emotions with Emotional DAN

Authentic Emotion Mapping: Benchmarking Facial Expressions in Real News

Acquisition and Analysis of Facial Electromyographic Signals for Emotion Recognition

Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning

Bi-Model Engagement Emotion Recognition Based on Facial and Upper-Body Landmarks and Machine Learning Approaches

The Influence of Emotional Facial Expression Intensity on Decoding Accuracy: High Intensity Does Not Yield High Accuracy

Expression Analysis Based on Face Regions in Read-world Conditions

Dissociable Roles of Internal Feelings and Face Recognition Ability in Facial Expression Decoding.

A Novel Active-Learning Based Emotion-Vision-Transformer Network for Expression Recognition