Abstract:Facial Expression Recognition (FER) is one of the most important topics in the domain of computer vision and pattern recognition, and it has attracted increasing attention for its scientific challenges and application potentials. In this article, we propose a novel and effective approach to FER using multi-model two-dimensional (2D) and 3D videos, which encodes both static and dynamic clues by scattering convolution network. First, a shape-based detection method is introduced to locate the start and the end of an expression in videos; segment its onset, apex, and offset states; and sample the important frames for emotion analysis. Second, the frames in Apex of 2D videos are represented by scattering, conveying static texture details. Those of 3D videos are processed in a similar way, but to highlight static shape details, several geometric maps in terms of multiple order differential quantities, i.e., Normal Maps and Shape Index Maps, are generated as the input of scattering, instead of original smooth facial surfaces. Third, the average of neighboring samples centred at each key texture frame or shape map in Onset is computed, and the scattering features extracted from all the average samples of 2D and 3D videos are then concatenated to capture dynamic texture and shape cues, respectively. Finally, Multiple Kernel Learning is adopted to combine the features in the 2D and 3D modalities and compute similarities to predict the expression label. Thanks to the scattering descriptor, the proposed approach not only encodes distinct local texture and shape variations of different expressions as by several milestone operators, such as SIFT, HOG, and so on, but also captures subtle information hidden in high frequencies in both channels, which is quite crucial to better distinguish expressions that are easily confused. The validation is conducted on the BU-4DFE and BP-4D databa ses, and the accuracies reached are very competitive, indicating its competency for this issue.

Learning Dynamics for Video Facial Expression Recognition

Joint Structured Sparsity Regularized Multiview Dimension Reduction for Video-Based Facial Expression Recognition.

Coarse-to-Fine Cascaded Networks with Smooth Predicting for Video Facial Expression Recognition

Automatic Analysis of Facial Expressions Based on Deep Covariance Trajectories

Video-Based Facial Expression Recognition Using Histogram Sequence of Local Gabor Binary Patterns from Three Orthogonal Planes

Facial Expression Recognition Based on the Fusion of Spatio-temporal Features in Video Sequences

Geometric-Convolutional Feature Fusion Based On Learning Propagation For Facial Expression Recognition

Rethinking the Learning Paradigm for Dynamic Facial Expression Recognition

The investigation of traditional models and machine learning models in dynamic facial expression recognition

SAANet: Siamese Action-Units Attention Network for Improving Dynamic Facial Expression Recognition

A Joint Local Spatial and Global Temporal CNN-Transformer for Dynamic Facial Expression Recognition

Observe Finer to Select Better: Learning Key Frame Extraction Via Semantic Coherence for Dynamic Facial Expression Recognition in the Wild

Learning facial expression and body gesture visual information for video emotion recognition

Automatic 4D Facial Expression Recognition Using DCT Features

Learning Expression Features via Deep Residual Attention Networks for Facial Expression Recognition From Video Sequences

Facial Expression Recognition Based On Group Domain Random Frame Extraction

Displaced Dynamic Expression Regression for Real-Time Facial Tracking and Animation

DPCNet: Dual Path Multi-Excitation Collaborative Network for Facial Expression Representation Learning in Videos

CFAN-SDA: Coarse-fine Aware Network with Static-Dynamic Adaptation for Facial Expression Recognition in Videos

Texture and Geometry Scattering Representation-Based Facial Expression Recognition in 2D+3D Videos

Magnifying Subtle Facial Motions for Effective 4D Expression Recognition