Abstract:Creating a large and natural facial expression database is a prerequisite for facial expression analysis and classification. It is, however, not only time consuming but also difficult to capture an adequately large number of spontaneous facial expression images and their meanings because no standard, uniform, and exact measurements are available for database collection and annotation. Thus, comprehensive first-hand data analyses of a spontaneous expression database may provide insight for future research on database construction, expression recognition, and emotion inference. This paper presents our analyses of a multimodal spontaneous facial expression database of natural visible and infrared facial expressions (NVIE). First, the effectiveness of emotion-eliciting videos in the database collection is analyzed with the mean and variance of the subjects' self-reported data. Second, an interrater reliability analysis of raters' subjective evaluations for apex expression images and sequences is conducted using Kappa and Kendall's coefficients. Third, we propose a matching rate matrix to explore the agreements between displayed spontaneous expressions and felt affective states. Lastly, the thermal differences between the posed and spontaneous facial expressions are analyzed using a paired-samples t-test. The results of these analyses demonstrate the effectiveness of our emotion-inducing experimental design, the gender difference in emotional responses, and the coexistence of multiple emotions/expressions. Facial image sequences are more informative than apex images for both expression and emotion recognition. Labeling an expression image or sequence with multiple categories together with their intensities could be a better approach than labeling the expression image or sequence with one dominant category. The results also demonstrate both the importance of facial expressions as a means of communication to convey affective states and the diversity of the displayed ma- ifestations of felt emotions. There are indeed some significant differences between the temperature difference data of most posed and spontaneous facial expressions, many of which are found in the forehead and cheek regions.

Non-contact Emotion Analysis Based on Videos.

Discrete Emotions Discovered by Contactless Measurement of Facial Blood Flows

Exploiting EEG signals and audiovisual feature fusion for video emotion recognition

Facial-Video-Based Physiological Signal Measurement: Recent Advances and Affective Applications

Classification of emotional states via transdermal cardiovascular spatiotemporal facial patterns using multispectral face videos

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

Sensing emotional valence and arousal dynamics through automated facial action unit analysis

Real-Time Movie-Induced Discrete Emotion Recognition from EEG Signals

Your blush gives you away: detecting hidden mental states with remote photoplethysmography and thermal imaging

Recognizing Emotions Based on Human Actions in Videos.

A Naturalistic Database of Thermal Emotional Facial Expressions and Effects of Induced Emotions on Memory

Construction and cross-correlation analysis of the affective physiological response database

A Noncontact Emotion Recognition Method Based on Complexion and Heart Rate

Analyses of a Multimodal Spontaneous Facial Expression Database

Physiological Sensors Based Emotion Recognition While Experiencing Tactile Enhanced Multimedia

Emotion Recognition Based on Skin Potential Signals with a Portable Wireless Device

Implicit Media Tagging and Affect Prediction from video of spontaneous facial expressions, recorded with depth camera

Recognition of Emotion According to the Physical Elements of the Video