Abstract:Affect recognition based on subjects' facial expressions has been a topic of major research in the attempt to generate machines that can understand the way subjects feel, act and react. In the past, due to the unavailability of large amounts of data captured in real-life situations, research has mainly focused on controlled environments. However, recently, social media and platforms have been widely used. Moreover, deep learning has emerged as a means to solve visual analysis and recognition problems. This paper exploits these advances and presents significant contributions for affect analysis and recognition in-the-wild. Affect analysis and recognition can be seen as a dual knowledge generation problem, involving: i) creation of new, large and rich in-the-wild databases and ii) design and training of novel deep neural architectures that are able to analyse affect over these databases and to successfully generalise their performance on other datasets. The paper focuses on large in-the-wild databases, i.e., Aff-Wild and Aff-Wild2 and presents the design of two classes of deep neural networks trained with these databases. The first class refers to uni-task affect recognition, focusing on prediction of the valence and arousal dimensional variables. The second class refers to estimation of all main behavior tasks, i.e. valence-arousal prediction; categorical emotion classification in seven basic facial expressions; facial Action Unit detection. A novel multi-task and holistic framework is presented which is able to jointly learn and effectively generalize and perform affect recognition over all existing in-the-wild databases. Large experimental studies illustrate the achieved performance improvement over the existing state-of-the-art in affect recognition.

Facial Affect Analysis: Learning from Synthetic Data & Multi-Task Learning Challenges

Multi-Task Learning for Emotion Descriptors Estimation at the fourth ABAW Challenge

Facial Affective Behavior Analysis Method for 5th ABAW Competition

Affective Behaviour Analysis via Progressive Learning

Facial Expression Recognition In-the-Wild with Deep Pre-trained Models.

Multi-Task Learning Framework for Emotion Recognition In-the-Wild

Multi-modal Facial Action Unit Detection with Large Pre-trained Models for the 5th Competition on Affective Behavior Analysis in-the-wild

An Effective Ensemble Learning Framework for Affective Behaviour Analysis

Affective Behaviour Analysis via Integrating Multi-Modal Knowledge

Facial Affect Recognition based on Transformer Encoder and Audiovisual Fusion for the ABAW5 Challenge

A Multi-term and Multi-task Analyzing Framework for Affective Analysis in-the-wild

Transformer-based Multimodal Information Fusion for Facial Expression Analysis

A Unified Approach to Facial Affect Analysis: the MAE-Face Visual Representation.

Multi-modal Facial Affective Analysis based on Masked Autoencoder

Facial Expression Recognition Based on Multi-modal Features for Videos in the Wild

Facial Emotion Recognition with Noisy Multi-task Annotations

Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework

A Multi-task Mean Teacher for Semi-supervised Facial Affective Behavior Analysis

Facial Affect Recognition based on Multi Architecture Encoder and Feature Fusion for the ABAW7 Challenge

Affective Behavior Analysis using Task-adaptive and AU-assisted Graph Network

Affective Expression Analysis in-the-wild using Multi-Task Temporal Statistical Deep Learning Model