Abstract:In the context of HCI, building an automatic system to recognize affect of human facial expression in real-world condition is very crucial to make machine interact naturallisticaly with a man. However, existing facial emotion databases usually contain expression in the limited scenario under well-controlled condition. Aff-Wild is currently the largest database consisting of spontaneous facial expression in the wild annotated with valence and arousal. The first contribution of this project is the completion of extending Aff-Wild database which is fulfilled by collecting videos from YouTube on which the videos have spontaneous facial expressions in the wild, annotating videos with valence and arousal ranging in [-1,1], detecting faces in frames using FFLD2 detector and partitioning the whole data set into train, validate and test set, with 527056, 94223 and 135145 frames. The diversity is guaranteed regarding age, ethnicity and values of valence and arousal. The ratio of male to female is close to 1. Regarding the techniques used to build the automatic system, deep learning is outstanding since almost all winning methods in emotion challenges adopt DNN techniques. The second contribution of this project is that an end-to-end DNN is constructed to have joint CNN and RNN block and gives the estimation on valence and arousal for each frame in sequential data. VGGFace, ResNet, DenseNet with the corresponding pre-trained model for CNN block and LSTM, GRU, IndRNN, Attention mechanism for RNN block are experimented aiming to find the best combination. Fine tuning and transfer learning techniques are also tried out. By comparing the CCC evaluation value on test data, the best model is found to be pre-trained VGGFace connected with 2 layers GRU with attention mechanism. The models test performance is 0.555 CCC for valence with sequence length 80 and 0.499 CCC for arousal with sequence length 70.

MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

HEU Emotion: A Large-scale Database for Multi-modal Emotion Recognition in the Wild

HEU Emotion: a Large-Scale Database for Multimodal Emotion Recognition in the Wild

4DME: A Spontaneous 4D Micro-Expression Dataset with Multimodalities

Aff-Wild Database and AffWildNet

DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild

Blended Emotion in-the-Wild: Multi-label Facial Expression Recognition Using Crowdsourced Annotations and Deep Locality Feature Learning

DFME: A New Benchmark for Dynamic Facial Micro-expression Recognition

Multimodal Adaptive Emotion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels

Compound Expression Recognition In-the-wild with AU-assisted Meta Multi-task Learning

Facial Expression Recognition Based on Multi-modal Features for Videos in the Wild

MEAD: A Large-Scale Audio-Visual Dataset for Emotional Talking-Face Generation

Building a Chinese Natural Emotional Audio-Visual Database

The design and evaluation of multimode affective database

Transformer-Based Multimodal Emotional Perception for Dynamic Facial Expression Recognition in the Wild

Evaluation and analysis of visual perception using attention-enhanced computation in multimedia affective computing

MPED: A Multi-Modal Physiological Emotion Database for Discrete Emotion Recognition

AffectNet+: A Database for Enhancing Facial Expression Recognition with Soft-Labels

Fine-Grained Facial Expression Recognition in the Wild.