Abstract:Human feelings expressed through verbal (e.g. voice) and non-verbal communication channels (e.g. face or body) can influence either human actions or interactions. In the literature, most of the attention was given to facial expressions for the analysis of emotions conveyed through non-verbal behaviors. Despite this, psychology highlights that the body is an important indicator of the human affective state in performing daily life activities. Therefore, this paper presents a novel method for affective action and interaction recognition from videos, exploiting multi-view representation learning and only full-body handcrafted characteristics selected following psychological and proxemic studies. Specifically, 2D skeletal data are extracted from RGB video sequences to derive diverse low-level skeleton features, i.e. multi-views, modeled through the bag-of-visual-words clustering approach generating a condition-related codebook. In this way, each affective action and interaction within a video can be represented as a frequency histogram of codewords. During the learning phase, for each affective class, training samples are used to compute its global histogram of codewords stored in a database and later used for the recognition task. In the recognition phase, the video frequency histogram representation is matched against the database of class histograms and classified as the closest affective class in terms of Euclidean distance. The effectiveness of the proposed system is evaluated on a specifically collected dataset containing 6 emotion for both actions and interactions, on which the proposed system obtains 93.64% and 90.83% accuracy, respectively. In addition, the devised strategy also achieves in line performances with other literature works based on deep learning when tested on a public collection containing 6 emotions plus a neutral state, demonstrating the effectiveness of the presented approach and confirming the findings in psychological and proxemic studies.

Recognizing Emotions Based on Human Actions in Videos.

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Emotion Recognition in Videos via Fusing Multimodal Features.

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Exploiting EEG signals and audiovisual feature fusion for video emotion recognition

Predicting Emotions in User-Generated Videos.

An Ensemble Approach for Facial Expression Analysis in Video

Emotion Detection from Speech to Enrich Multimedia Content

Human Action Recognition Using Deep Learning Methods.

Automatic Recognition of Facial Displays of Unfelt Emotions

Happy Index - Analysis Based on Automatic Recognition of Emotion Flow.

A robust multimodal approach for emotion recognition

Audio-visual Based Emotion Recognition-a New Approach

Beyond Facial Expressions: Learning Human Emotion from Body Gestures.

Detection of (Hidden) Emotions from Videos using Muscles Movements and Face Manifold Embedding

Emotion Recognition from the perspective of Activity Recognition

Implicit Media Tagging and Affect Prediction from video of spontaneous facial expressions, recorded with depth camera

Affective Action and Interaction Recognition by Multi-View Representation Learning from Handcrafted Low-Level Skeleton Features

Recognition of Emotion According to the Physical Elements of the Video

Emotional Tagging Of Videos By Exploring Multiple Emotions' Coexistence

Implicit Video Emotion Tagging from Audiences’ Facial Expression