Abstract:Facial Expression Recognition (FER) is one of the most important topics in the domain of computer vision and pattern recognition, and it has attracted increasing attention for its scientific challenges and application potentials. In this article, we propose a novel and effective approach to FER using multi-model two-dimensional (2D) and 3D videos, which encodes both static and dynamic clues by scattering convolution network. First, a shape-based detection method is introduced to locate the start and the end of an expression in videos; segment its onset, apex, and offset states; and sample the important frames for emotion analysis. Second, the frames in Apex of 2D videos are represented by scattering, conveying static texture details. Those of 3D videos are processed in a similar way, but to highlight static shape details, several geometric maps in terms of multiple order differential quantities, i.e., Normal Maps and Shape Index Maps, are generated as the input of scattering, instead of original smooth facial surfaces. Third, the average of neighboring samples centred at each key texture frame or shape map in Onset is computed, and the scattering features extracted from all the average samples of 2D and 3D videos are then concatenated to capture dynamic texture and shape cues, respectively. Finally, Multiple Kernel Learning is adopted to combine the features in the 2D and 3D modalities and compute similarities to predict the expression label. Thanks to the scattering descriptor, the proposed approach not only encodes distinct local texture and shape variations of different expressions as by several milestone operators, such as SIFT, HOG, and so on, but also captures subtle information hidden in high frequencies in both channels, which is quite crucial to better distinguish expressions that are easily confused. The validation is conducted on the BU-4DFE and BP-4D databa ses, and the accuracies reached are very competitive, indicating its competency for this issue.

Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework

Recognising Spontaneous Facial Micro-Expressions

Spontaneous Versus Posed Smile Recognition Using Discriminative Local Spatial-Temporal Descriptors

Automatic facial expression recognition on a single 3D face by exploring shape deformation.

Automatic facial expression recognition based on spatiotemporal descriptors

Convolutional Neural Networks for Posed and Spontaneous Expression Recognition

Discriminative Spatiotemporal Local Binary Pattern with Revisited Integral Projection for Spontaneous Facial Micro-Expression Recognition

Posed and Spontaneous Expression Recognition Through Modeling Their Spatial Patterns

Fully automatic 3D facial expression recognition using a region-based approach

Capturing Spatial and Temporal Patterns for Distinguishing Between Posed and Spontaneous Expressions

Three-dimensional face recognition under expression variation

Spontaneous Facial Micro-Expression Recognition using Discriminative Spatiotemporal Local Binary Pattern with an Improved Integral Projection

Spontaneous Versus Posed Smile Recognition Via Region-Specific Texture Descriptor and Geometric Facial Dynamics

Real-Time Facial Expression Recognition Based on Attention Mechanism

Posed and spontaneous facial expression differentiation using deep Boltzmann machines

Facial Expression Recognition Based on Spatio-Temporal Interest Points for Depth Sequences

Facial Expression Recognition Based on the Fusion of Spatio-temporal Features in Video Sequences

Learning Transferable Sparse Representations for Cross-corpus Facial Expression Recognition

Posed and spontaneous expression distinguishment from infrared thermal images

Spontaneous Facial Expression Recognition by Using Feature-Level Fusion of Visible and Thermal Infrared Images

Texture and Geometry Scattering Representation-Based Facial Expression Recognition in 2D+3D Videos