Abstract:Current works on facial action unit (AU) recognition typically require fully AU-labeled training samples. To reduce the reliance on time-consuming manual AU annotations, we propose a novel semi-supervised AU recognition method leveraging two kinds of readily available auxiliary information. The method leverages the dependencies between AUs and expressions as well as the dependencies among AUs, which are caused by facial anatomy and therefore embedded in all facial images, independent on their AU annotation status. The other auxiliary information is facial image synthesis given AUs, the dual task of AU recognition from facial images, and therefore has intrinsic probabilistic connections with AU recognition, regardless of AU annotations. Specifically, we propose a dual semi-supervised generative adversarial network for AU recognition from partially AU-labeled and fully expression-labeled facial images. The proposed network consists of an AU classifier C, an image generator G , and a discriminator D. In addition to minimize the supervised losses of the AU classifier and the face generator for labeled training data, we explore the probabilistic duality between the tasks using adversary learning to force the convergence of the face-AU-expression tuples generated from the AU classifier and the face generator, and the ground-truth distribution in labeled data for all training data. This joint distribution also includes the inherent AU dependencies. Furthermore, we reconstruct the facial image using the output of the AU classifier as the input of the face generator, and create AU labels by feeding the output of the face generator to the AU classifier. We minimize reconstruction losses for all training data, thus exploiting the informative feedback provided by the dual tasks. Within-database and cross-database experiments on three benchmark databases demonstrate the superiority of our method in both AU recognition and face synthesis compared to state-of-the-art works.

AU (action Unit) Detection Based on BEGAN Data Augmentation

Facial Action Units Detection Aided by Global-Local Expression Embedding

AU Data Augmentation Method Based on Generative Adversarial Networks

Action Unit Driven Facial Expression Synthesis from a Single Image with Patch Attentive GAN.

Detecting Facial Action Units from Global-Local Fine-grained Expressions

FG-Net: Facial Action Unit Detection with Generalizable Pyramidal Features

Personalized Multiple Facial Action Unit Recognition through Generative Adversarial Recognition Network.

An Improved Generative Adversarial Network for Micro-expressions Based on Multi-label Learning from Action Units.

Learning facial expression-aware global-to-local representation for robust action unit detection

AU-aware Deep Networks for Facial Expression Recognition

Expression-assisted Facial Action Unit Recognition under Incomplete AU Annotation.

Global-to-local Expression-aware Embeddings for Facial Action Unit Detection

Dual Semi-Supervised Learning for Facial Action Unit Recognition.

AU-inspired Deep Networks for Facial Expression Feature Learning

AU-aware Graph Convolutional Network for Macroand Micro-expression Spotting.

Region Based Adversarial Synthesis of Facial Action Units

Semi Supervised Unconstrained Action Unit Detection via Latent Feature Domain

Region And Temporal Dependency Fusion For Multi-Label Action Unit Detection

AU R-CNN: Encoding Expert Prior Knowledge into R-CNN for Action Unit Detection

Conditional Adversarial Synthesis of 3D Facial Action Units.

Dual Learning for Facial Action Unit Detection under Nonfull Annotation.