Abstract:Facial action unit (AU) recognition has been applied in a wild range of fields, and has attracted great attention in the past two decades. Most existing works on AU recognition assumed that the complete label assignment for each training image is available, which is often not the case in practice. Labeling AU is expensive and time consuming process. Moreover, due to the AU ambiguity and subjective difference, some AUs are difficult to label reliably and confidently. Many AU recognition works try to train the classifier for each AU independently, which is of high computation cost and ignores the dependency among different AUs. In this work, we formulate AU recognition under incomplete data as a multi-label learning with missing labels (MLML) problem. Most existing MLML methods usually employ the same features for all classes. However, we find this setting is unreasonable in AU recognition, as the occurrence of different AUs produce changes of skin surface displacement or face appearance in different face regions. If using the shared features for all AUs, much noise will be involved due to the occurrence of other AUs. Consequently, the changes of the specific AUs cannot be clearly highlighted, leading to the performance degradation. Instead, we propose to extract the most discriminative features for each AU individually, which are learned by the supervised learning method. The learned features are further embedded into the instance-level label smoothness term of our model, which also includes the label consistency and the class-level label smoothness. Both a global solution using st-cut and an approximated solution using conjugate gradient (CG) descent are provided. Experiments on both posed and spontaneous facial expression databases demonstrate the superiority of the proposed method in comparison with several state-of-the-art works.

Multi-Label Class Balancing Algorithm for Action Unit Detection

Unique Class Group Based Multi-Label Balancing Optimizer for Action Unit Detection

Multi-label Learning with Missing Values using Combined Facial Action Unit Datasets

Joint Patch And Multi-Label Learning For Facial Action Unit Detection

Fighting noise and imbalance in Action Unit detection problems

An Attention-based Method for Action Unit Detection at the 3rd ABAW Competition

Handling Missing Labels and Class Imbalance Challenges Simultaneously for Facial Action Unit Recognition

Multi-Task Learning for Emotion Descriptors Estimation at the fourth ABAW Challenge

Multi-modal Facial Action Unit Detection with Large Pre-trained Models for the 5th Competition on Affective Behavior Analysis in-the-wild

Feature and Label Relation Modeling for Multiple-Facial Action Unit Classification and Intensity Estimation

Multi-modal Multi-label Facial Action Unit Detection with Transformer

Facial Action Unit Recognition under Incomplete Data Based on Multi-Label Learning with Missing Labels.

Semi Supervised Unconstrained Action Unit Detection via Latent Feature Domain

Facial Action Unit Detection with Local Key Facial Sub-region Based Multi-label Classification for Micro-expression Analysis.

Facial Affect Analysis: Learning from Synthetic Data & Multi-Task Learning Challenges

CCT: A Cross-Concat and Temporal Neural Network for Multi-Label Action Unit Detection

ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges

Multiple-Facial Action Unit Recognition by Shared Feature Learning and Semantic Relation Modeling

Deep Facial Action Unit Recognition and Intensity Estimation from Partially Labelled Data

Dynamic Cascades with Bidirectional Bootstrapping for Action Unit Detection in Spontaneous Facial Behavior

ABAW : Facial Expression Recognition in the wild