Abstract:Current facial action unit (AU) recognition typically includes supervised training, where the fully AU annotated training images are required. Due to the nuances of facial appearance and individual differences, AU annotation is a time-consuming, expensive, and error-prone process. Facial expression is relatively simple to label, since facial expressions describe facial behavior globally and the number of expressions appearing on a face is much less than that of AUs. Furthermore, there exist strong dependencies between AUs and expressions, referred to as domain knowledge. Such domain knowledge is inherent in facial anatomy and facial behavior. Therefore, in this paper, we propose a novel weakly supervised AU recognition method to jointly learn multiple AU classifiers with expression annotations but without any AU annotations by leveraging domain knowledge. Specifically, we first summarize the expression-dependent AU ranking from the domain knowledge of conditional probabilities of AUs given expressions. Then, we formulate the weakly supervised AU recognition as a multilabel ranking problem and propose an efficient learning algorithm to solve it. Furthermore, we extend the proposed weakly supervised AU recognition method to a semi-supervised learning scenario when partial AU labeled samples are available. Experimental results on three benchmark databases demonstrate that the proposed method can successfully exploit domain knowledge for multiple AU recognition and, thus, outperforms both state-of-the-art weakly supervised AU recognition method and the semi-supervised AU recognition method.

Facial Action Unit Detection by Exploring the Weak Relationships Between AU Labels

Facial Action Units Detection Aided by Global-Local Expression Embedding

Facial Action Unit Detection Using Attention and Relation Learning

Facial Action Unit Detection Via Adaptive Attention and Relation.

Region And Temporal Dependency Fusion For Multi-Label Action Unit Detection

Facial Action Unit Detection with Emotion Consistency: a Cross-Modal Learning Approach

Weakly-Supervised Attention and Relation Learning for Facial Action Unit Detection

Facial Action Unit Detection with Multilayer Fused Multi-Task and Multi-Label Deep Learning Network

Weakly-Supervised Unconstrained Action Unit Detection Via Feature Disentanglement

Learning facial expression-aware global-to-local representation for robust action unit detection

Learning Contrastive Feature Representations for Facial Action Unit Detection

Semi Supervised Unconstrained Action Unit Detection via Latent Feature Domain

Detecting Facial Action Units from Global-Local Fine-grained Expressions

Weakly Supervised Facial Action Unit Recognition with Domain Knowledge

Action Unit Detection with Joint Adaptive Attention and Graph Relation

Local Region Perception and Relationship Learning Combined with Feature Fusion for Facial Action Unit Detection

Cross-Modal Representation Learning For Lightweight And Accurate Facial Action Unit Detection

Action unit analysis enhanced facial expression recognition by deep neural network evolution

AU-aware Deep Networks for Facial Expression Recognition

Facial Action Unit Recognition Based on Self-Attention Spatiotemporal Fusion

Deep Facial Action Unit Recognition from Partially Labeled Data.