Abstract:Facial Expression Recognition (FER) is one of the most important topics in the domain of computer vision and pattern recognition, and it has attracted increasing attention for its scientific challenges and application potentials. In this article, we propose a novel and effective approach to FER using multi-model two-dimensional (2D) and 3D videos, which encodes both static and dynamic clues by scattering convolution network. First, a shape-based detection method is introduced to locate the start and the end of an expression in videos; segment its onset, apex, and offset states; and sample the important frames for emotion analysis. Second, the frames in Apex of 2D videos are represented by scattering, conveying static texture details. Those of 3D videos are processed in a similar way, but to highlight static shape details, several geometric maps in terms of multiple order differential quantities, i.e., Normal Maps and Shape Index Maps, are generated as the input of scattering, instead of original smooth facial surfaces. Third, the average of neighboring samples centred at each key texture frame or shape map in Onset is computed, and the scattering features extracted from all the average samples of 2D and 3D videos are then concatenated to capture dynamic texture and shape cues, respectively. Finally, Multiple Kernel Learning is adopted to combine the features in the 2D and 3D modalities and compute similarities to predict the expression label. Thanks to the scattering descriptor, the proposed approach not only encodes distinct local texture and shape variations of different expressions as by several milestone operators, such as SIFT, HOG, and so on, but also captures subtle information hidden in high frequencies in both channels, which is quite crucial to better distinguish expressions that are easily confused. The validation is conducted on the BU-4DFE and BP-4D databa ses, and the accuracies reached are very competitive, indicating its competency for this issue.

Collaborative Discriminative Multi-Metric Learning for Facial Expression Recognition in Video

Emotion Recognition in Videos via Fusing Multimodal Features.

Face Recognition Based on LDA and Improved Pairwise-Constrained Multiple Metric Learning Method.

Multi-threshold Deep Metric Learning for Facial Expression Recognition

Joint Structured Sparsity Regularized Multiview Dimension Reduction for Video-Based Facial Expression Recognition.

Collaborative Multi-View Metric Learning For Visual Classification

Combining 2D Gabor and Local Binary Pattern for Facial Expression Recognition Using Extreme Learning Machine

A Scalable Metric Learning-Based Voting Method for Expression Recognition

Dynamic Multi-Channel Metric Network for Joint Pose-Aware and Identity-Invariant Facial Expression Recognition

Attention-Set based Metric Learning for Video Face Recognition

Multi-View Exclusive Unsupervised Dimension Reduction for Video-Based Facial Expression Recognition

Texture and Geometry Scattering Representation-Based Facial Expression Recognition in 2D+3D Videos

Face Recognition Based on Local Binary Pattern and Improved Pairwise-constrained Multiple Metric Learning

Transfer Model Collaborating Metric Learning and Dictionary Learning for Cross-Domain Facial Expression Recognition.

Multi-manifold Metric Learning for Face Recognition Based on Image Sets

Discriminative Deep Multi-Task Learning for Facial Expression Recognition.

Identity-aware Facial Expression Recognition via Deep Metric Learning based on Synthesized Images

3D Facial Expression Recognition Via Multiple Kernel Learning of Multi-Scale Local Normal Patterns

Discriminative Multimetric Learning for Kinship Verification

Multi-view metric learning for visual classification

Twin attention based multi-task convolutional bidirectional long short term memory for facial expression recognition