Abstract:Facial action unit (AU) recognition has been applied in a wild range of fields, and has attracted great attention in the past two decades. Most existing works on AU recognition assumed that the complete label assignment for each training image is available, which is often not the case in practice. Labeling AU is expensive and time consuming process. Moreover, due to the AU ambiguity and subjective difference, some AUs are difficult to label reliably and confidently. Many AU recognition works try to train the classifier for each AU independently, which is of high computation cost and ignores the dependency among different AUs. In this work, we formulate AU recognition under incomplete data as a multi-label learning with missing labels (MLML) problem. Most existing MLML methods usually employ the same features for all classes. However, we find this setting is unreasonable in AU recognition, as the occurrence of different AUs produce changes of skin surface displacement or face appearance in different face regions. If using the shared features for all AUs, much noise will be involved due to the occurrence of other AUs. Consequently, the changes of the specific AUs cannot be clearly highlighted, leading to the performance degradation. Instead, we propose to extract the most discriminative features for each AU individually, which are learned by the supervised learning method. The learned features are further embedded into the instance-level label smoothness term of our model, which also includes the label consistency and the class-level label smoothness. Both a global solution using st-cut and an approximated solution using conjugate gradient (CG) descent are provided. Experiments on both posed and spontaneous facial expression databases demonstrate the superiority of the proposed method in comparison with several state-of-the-art works.

Multiple Facial Action Unit Recognition by Learning Joint Features and Label Relations.

Capturing Feature and Label Relations Simultaneously for Multiple Facial Action Unit Recognition

Multiple Facial Action Unit Recognition Enhanced by Facial Expressions.

Feature and Label Relation Modeling for Multiple-Facial Action Unit Classification and Intensity Estimation

Facial Action Unit Recognition and Intensity Estimation Enhanced Through Label Dependencies

Deep Facial Action Unit Recognition from Partially Labeled Data.

Multiple-Facial Action Unit Recognition by Shared Feature Learning and Semantic Relation Modeling

Deep Region And Multi-Label Learning For Facial Action Unit Detection

Capturing Global Semantic Relationships for Facial Action Unit Recognition

Joint Patch And Multi-Label Learning For Facial Action Unit Detection

Dual Learning for Joint Facial Landmark Detection and Action Unit Recognition

Capture expression-dependent AU relations for expression recognition

Multi-scale Dynamic and Hierarchical Relationship Modeling for Facial Action Units Recognition

Facial Action Unit Detection Using Attention and Relation Learning

Joint Patch and Multi-label Learning for Facial Action Unit and Holistic Expression Recognition

Region And Temporal Dependency Fusion For Multi-Label Action Unit Detection

Facial Action Unit Recognition under Incomplete Data Based on Multi-Label Learning with Missing Labels.

Exploring Domain Knowledge for Facial Expression-Assisted Action Unit Activation Recognition.

Attention Based Relation Network for Facial Action Units Recognition

Facial Action Unit recognition by relation modeling from both qualitative knowledge and quantitative data

Multi-Scale Region with Local Relationship Learning for Facial Action Unit Detection