Abstract:We propose to learn a cascade of globally-optimized modular boosted ferns (GoMBF) to solve multi-modal facial motion regression for real-time 3D facial tracking from a monocular RGB camera. GoMBF is a deep composition of multiple regression models with each is a boosted ferns initially trained to predict partial motion parameters of the same modality, and then concatenated together via a global optimization step to form a singular strong boosted ferns that can effectively handle the whole regression target. It can explicitly cope with the modality variety in output variables, while manifesting increased fitting power and a faster learning speed comparing against the conventional boosted ferns. By further cascading a sequence of GoMBFs (GoMBF-Cascade) to regress facial motion parameters, we achieve competitive tracking performance on a variety of in-the-wild videos comparing to the state-of-the-art methods which either have higher computational complexity or require much more training data. It provides a robust and highly elegant solution to real-time 3D facial tracking using a small set of training data and hence makes it more practical in real-world applications. We further deeply investigate the effect of synthesized facial images on training non-deep learning methods such as GoMBF-Cascade for 3D facial tracking. We apply three types synthetic images with various naturalness levels for training two different tracking methods, and compare the performance of the tracking models trained on real data, on synthetic data and on a mixture of data. The experimental results indicate that, i) the model trained purely on synthetic facial imageries can hardly generalize well to unconstrained real-world data, ii) involving synthetic faces into training benefits tracking in some certain scenarios but degrades the tracking model's generalization ability. These two insights could benefit a range of non-deep learning facial image analysis tasks where the labelled real data is difficult to acquire.

Facial landmark localization based on hierarchical pose regression with cascaded random ferns.

Hierarchical Facial Landmark Localization Via Cascaded Random Binary Patterns

A Cross-Dimension Annotations Method for 3D Structural Facial Landmark Extraction

Cascade of Forests for Face Alignment

FDN: Feature Decoupling Network for Head Pose Estimation.

Robust facial landmark localization using classified random ferns and pose-based initialization.

Multiview Facial Landmark Localization in RGB-D Images Via Hierarchical Regression with Binary Patterns.

Joint Head Pose and Facial Landmark Regression from Depth Images

Robust Facial Landmark Localization Based on Two-Stage Cascaded Pose Regression

Cross-Cascading Regression for Simultaneous Head Pose Estimation and Facial Landmark Detection.

Coupled Cascade Regression For Simultaneous Facial Landmark Detection And Head Pose Estimation

Facial Landmark Localization in the Wild by Backbone-Branches Representation Learning

Real-time Localization of 3D Facial Landmarks

Facial Landmark Detection under Large Pose

Coupled cascade regression from real and synthesized faces for simultaneous landmark detection and head pose estimation

Real-time 3D Facial Tracking Via Cascaded Compositional Learning.

On 3D Face Reconstruction via Cascaded Regression in Shape Space

Facial Landmark Machines: A Backbone-Branches Architecture with Progressive Representation Learning

Robust Facial Landmark Detection Via Heatmap-Offset Regression

Dual Sparse Constrained Cascade Regression for Robust Face Alignment.

Accurate Landmarking from 3D Facial Scans by CNN and Cascade Regression.