Abstract:Automated facial expression analysis from image sequences for continuous emotion recognition is a very challenging task due to the loss of the three-dimensional information during the image formation process. State-of-the-art relied on estimating dynamic textures features and convolutional neural network features to derive spatio-temporal features. Despite their great success, such features are insensitive to micro facial muscle deformations and are affected by identity, face pose, illumination variation, and self-occlusion. In this work, we argue that retrieving, from image sequences, 3D facial spatio-temporal information, which describes the natural facial muscle deformation, provides a semantical and efficient way of representation and is useful for emotion recognition. In this paper, we propose a framework for extracting three-dimensional facial spatio-temporal features from monocular image sequences using an extended 3D Morphable Model (3DMM) which disentangles the identity factor from the facial expressions of a specific person. An LSTM model is used to evaluate the effectiveness of the proposed spatio-temporal features on video-based facial expression recognition task and continuous affect recognition task. Experimental results, on the AFEW6.0 datasets for facial expression recognition, and the RECOLA and SEMAINE datasets for continuous emotion prediction, illustrate the potential of the proposed 3D spatio-temporal features for facial expressions analysis and continuous affect recognition, as well as their efficiency compared to recent state-of-the-art features.

Efficient Modeling of Long Temporal Contexts for Continuous Emotion Recognition.

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Learning Long-Term Temporal Contexts Using Skip RNN for Continuous Emotion Recognition

Learning Hierarchical Emotion Context for Continuous Dimensional Emotion Recognition from Video Sequences

A multimodal fusion-based deep learning framework combined with local-global contextual TCNs for continuous emotion recognition from videos

Continuous Multimodal Emotion Prediction Based on Long Short Term Memory Recurrent Neural Network

Time-Delay Neural Network for Continuous Emotional Dimension Prediction from Facial Expression Sequences.

Emotion Recognition via Environmental Context and Human Body

End-to-End Continuous Emotion Recognition from Video Using 3D Convlstm Networks

Multi-scale Temporal Modeling for Dimensional Emotion Recognition in Video

An Emotion-Embedded Visual Attention Model for Dimensional Emotion Context Learning.

Multi-modal Emotion Recognition with Temporal-Band Attention Based on LSTM-RNN.

Branch-Fusion-Net for Multi-Modal Continuous Dimensional Emotion Recognition

Facial Micro-expression Recognition Using Enhanced Temporal Feature-Wise Model

Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks

Monocular 3D Facial Expression Features for Continuous Affect Recognition

Residual multimodal Transformer for expression‐EEG fusion continuous emotion recognition

Transformer Encoder With Multi-Modal Multi-Head Attention for Continuous Affect Recognition

A Two-Stage Spatiotemporal Attention Convolution Network for Continuous Dimensional Emotion Recognition from Facial Video

Multimodal Continuous Emotion Recognition with Data Augmentation Using Recurrent Neural Networks

Context-dependent Emotion Recognition.