Abstract:Most human emotion recognition methods largely depend on classifying stereotypical facial expressions that represent emotions. However, such facial expressions do not necessarily correspond to actual emotional states and may correspond to communicative intentions. In other cases, emotions are hidden, cannot be expressed, or may have lower arousal manifested by less pronounced facial expressions, as may occur during passive video viewing. This study improves an emotion classification approach developed in a previous study, which classifies emotions remotely without relying on stereotypical facial expressions or contact-based methods, using short facial video data. In this approach, we desire to remotely sense transdermal cardiovascular spatiotemporal facial patterns associated with different emotional states and analyze this data via machine learning. In this paper, we propose several improvements, which include a better remote heart rate estimation via a preliminary skin segmentation, improvement of the heartbeat peaks and troughs detection process, and obtaining a better emotion classification accuracy by employing an appropriate deep learning classifier using an RGB camera input only with data. We used the dataset obtained in the previous study, which contains facial videos of 110 participants who passively viewed 150 short videos that elicited the following five emotion types: amusement, disgust, fear, sexual arousal, and no emotion, while three cameras with different wavelength sensitivities (visible spectrum, near-infrared, and longwave infrared) recorded them simultaneously. From the short facial videos, we extracted unique high-resolution spatiotemporal, physiologically affected features and examined them as input features with different deep-learning approaches. An EfficientNet-B0 model type was able to classify participants' emotional states with an overall average accuracy of 47.36% using a single input spatiotemporal feature map obtained from a regular RGB camera.

Classifying and Visualizing Emotions with Emotional DAN

Recognizing Emotions with EmotionalDAN

I Know How You Feel: Emotion Recognition with Facial Landmarks

Classifying Emotions and Engagement in Online Learning Based on a Single Facial Expression Recognition Neural Network

Deep Learning Models for Automated Classification of Dog Emotional States from Facial Expressions

Deep Learning Based Human Emotion Exposure Detection Using Vocal and Countenance

Human Face Emotion in 3D Using Machine Learning

EMOCA: Emotion Driven Monocular Face Capture and Animation

Facial Emotion Detection Using Convolutional Neural Networks and Representational Autoencoder Units

Human Facial Emotion Recognition using Deep Neural Networks

Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning

Can CNNs Accurately Classify Human Emotions? A Deep-Learning Facial Expression Recognition Study

Emotion Recognition for Challenged People Facial Appearance in Social using Neural Network

Facial Emotions Recognition Using Deep Learning Technology

Human emotion recognition by analyzing facial expressions, heart rate and blogs using deep learning method

Distract Your Attention: Multi-Head Cross Attention Network for Facial Expression Recognition

Emotion Detection through Body Gesture and Face

Facial Expression Analysis - A Hybrid Neural Network Based Approach

A real time face emotion classification and recognition using deep learning model

EMONET: A Cross Database Progressive Deep Network for Facial Expression Recognition

Masked Face Emotion Recognition Based on Facial Landmarks and Deep Learning Approaches for Visually Impaired People