Abstract:Facial Expression Recognition (FER) is one of the most important topics in the domain of computer vision and pattern recognition, and it has attracted increasing attention for its scientific challenges and application potentials. In this article, we propose a novel and effective approach to FER using multi-model two-dimensional (2D) and 3D videos, which encodes both static and dynamic clues by scattering convolution network. First, a shape-based detection method is introduced to locate the start and the end of an expression in videos; segment its onset, apex, and offset states; and sample the important frames for emotion analysis. Second, the frames in Apex of 2D videos are represented by scattering, conveying static texture details. Those of 3D videos are processed in a similar way, but to highlight static shape details, several geometric maps in terms of multiple order differential quantities, i.e., Normal Maps and Shape Index Maps, are generated as the input of scattering, instead of original smooth facial surfaces. Third, the average of neighboring samples centred at each key texture frame or shape map in Onset is computed, and the scattering features extracted from all the average samples of 2D and 3D videos are then concatenated to capture dynamic texture and shape cues, respectively. Finally, Multiple Kernel Learning is adopted to combine the features in the 2D and 3D modalities and compute similarities to predict the expression label. Thanks to the scattering descriptor, the proposed approach not only encodes distinct local texture and shape variations of different expressions as by several milestone operators, such as SIFT, HOG, and so on, but also captures subtle information hidden in high frequencies in both channels, which is quite crucial to better distinguish expressions that are easily confused. The validation is conducted on the BU-4DFE and BP-4D databa ses, and the accuracies reached are very competitive, indicating its competency for this issue.

Hierarchical Bayesian Theme Models for Multipose Facial Expression Recognition

Facial Expression Recognition Based on Auxiliary Models

Person-independent Facial Expression Recognition Via Hierarchical Classification

A Hierarchical Probabilistic Model for Facial Feature Detection

Texture and Geometry Scattering Representation-Based Facial Expression Recognition in 2D+3D Videos

Facial Expression Recognition Using Hierarchical Features with Deep Comprehensive Multipatches Aggregation Convolutional Neural Networks

Facial Expression Recognition Method Based on Multi-scale Detail Enhancement

HiT-MST: Dynamic Facial Expression Recognition with Hierarchical Transformers and Multi-Scale Spatiotemporal Aggregation.

An Enhanced Facial Expression Recognition Method Based on Multi-task Learning

A Spatio-Temporal Integrated Model Based on Local and Global Features for Video Expression Recognition

Dynamic Multi-Channel Metric Network for Joint Pose-Aware and Identity-Invariant Facial Expression Recognition

Facial Expression Recognition Based on Deep Evolutional Spatial-Temporal Networks

Muscular Movement Model Based Automatic 3D Facial Expression Recognition.

Multi-Attention Module for Dynamic Facial Emotion Recognition

Deep Global Multiple-Scale and Local Spatial-Channel Attention Dual-Branch Network for Pose-Invariant Facial Expression Recognition

Pose-adaptive Hierarchical Attention Network for Facial Expression Recognition

Simultaneous Facial Feature Tracking and Facial Expression Recognition.

Hierarchical Spatial Pyramidal Feature Representation Learning for Facial Expression Recognition

Adaptive Facial Expression Recognition Method Based on MBP and HMOG Feature

A hierarchical algorithm with multi-feature fusion for facial expression recognition