Abstract:In this paper, we examine 3 important issues in the practical use of state-of-the-art facial landmark detectors and show how a combination of specific architectural modifications can directly improve their accuracy and temporal stability. First, many facial landmark detectors require face normalization as a preprocessing step, which is accomplished by a separately-trained neural network that crops and resizes the face in the input image. There is no guarantee that this pre-trained network performs the optimal face normalization for landmark detection. We instead analyze the use of a spatial transformer network that is trained alongside the landmark detector in an unsupervised manner, and jointly learn optimal face normalization and landmark detection. Second, we show that modifying the output head of the landmark predictor to infer landmarks in a canonical 3D space can further improve accuracy. To convert the predicted 3D landmarks into screen-space, we additionally predict the camera intrinsics and head pose from the input image. As a side benefit, this allows to predict the 3D face shape from a given image only using 2D landmarks as supervision, which is useful in determining landmark visibility among other things. Finally, when training a landmark detector on multiple datasets at the same time, annotation inconsistencies across datasets forces the network to produce a suboptimal average. We propose to add a semantic correction network to address this issue. This additional lightweight neural network is trained alongside the landmark detector, without requiring any additional supervision. While the insights of this paper can be applied to most common landmark detectors, we specifically target a recently-proposed continuous 2D landmark detector to demonstrate how each of our additions leads to meaningful improvements over the state-of-the-art on standard benchmarks.

Attention-Based Accurate and Robust Facial Landmark Detector

Real-Time Facial Landmark Detection by Attention-driven Lightweight Network

A Cascade Face Spoofing Detector Based on Face Anti-Spoofing R-CNN and Improved Retinex LBP

Robust and Precise Facial Landmark Detection by Self-Calibrated Pose Attention Network

End-to-End Spatial Transform Face Detection and Recognition

Combining Data-driven and Model-driven Methods for Robust Facial Landmark Detection

A Robust Facial Landmark Detector with Mixed Loss

Robust Facial Landmark Detection Via Heatmap-Offset Regression

3-D Facial Landmarks Detection for Intelligent Video Systems

Deep Coupling Neural Network for Robust Facial Landmark Detection

Cascaded Structure-Learning Network with Using Adversarial Training for Robust Facial Landmark Detection

A Detailed Look At CNN-based Approaches In Facial Landmark Detection

FreeEnricher: Enriching Face Landmarks Without Additional Cost

Facial Component-Landmark Detection with Weakly-Supervised LR-CNN.

Towards Unconstrained Facial Landmark Detection Robust to Diverse Cropping Manners.

Facial Landmark Detection Via Cascade Multi-Channel Convolutional Neural Network

PFLD: A Practical Facial Landmark Detector.

Facial Landmark Localization by Enhanced Convolutional Neural Network.

Stacked Attention Hourglass Network Based Robust Facial Landmark Detection

Infinite 3D Landmarks: Improving Continuous 2D Facial Landmark Detection

Robust and Flexible Landmarks Detection for Uncontrolled Frontal Faces in the Wild