Abstract:Facial expressions can be represented largely by the dynamic variations of important facial expression parts, i.e., eyebrows, eyes, nose, and mouth. The features of these parts are regarded as local features. However, facial global information is also useful for recognition because it is a necessary complement to local features. In this paper, a spatio-temporal integrated model that jointly learns local and global features is proposed for video expression recognition. Firstly, to capture the action of facial key units, a spatio-temporal attention part-gradient-based hierarchical bidirectional recurrent neural network (spatio-temporal attention PGHRNN) is constructed. It can capture the dynamic variations of gradients around facial landmark points. In addition, a new kind of spatial attention mechanism is introduced to recalibrate the features of facial various parts adaptively. Secondly, to complement the local features extracted by the spatio-temporal attention PGHRNN, a squeeze-and-excitation residual network of 50 layers with long short-term memory network (SE-ResNet-50-LSTM) is used as a global feature extractor and classifier. Finally, to integrate the local and global features and improve the performance of facial expression recognition, a joint adaptive fine-tuning method (JAFTM) is proposed to combine the two networks, which can adaptively adjust the network weights. Extensive experiments demonstrate that our proposed model can achieve a superior recognition accuracy of 98.95% on CK + for 7-class facial expressions and 85.40% on MMI database, which outperforms other state-of-the-art methods.

Capturing Complex Spatio-temporal Relations among Facial Muscles for Facial Expression Recognition

Capturing Spatial and Temporal Patterns for Distinguishing Between Posed and Spontaneous Expressions

A Novel Dynamic Model Capturing Spatial and Temporal Patterns for Facial Expression Analysis

Automatic facial expression recognition based on spatiotemporal descriptors

Facial Expression Recognition Through Modeling Age-Related Spatial Patterns.

Simultaneous Facial Feature Tracking and Facial Expression Recognition.

Facial Micro-expression Recognition Using Enhanced Temporal Feature-Wise Model

A Spatio-Temporal Integrated Model Based on Local and Global Features for Video Expression Recognition

Real-Time facial expression recognition system based on HMM and feature point localization

A Facial Expression Model For Human-Like Agent

A Joint Local Spatial and Global Temporal CNN-Transformer for Dynamic Facial Expression Recognition

Facial Expression Recognition Method Based on a Part-Based Temporal Convolutional Network with a Graph-Structured Representation.

Facial Expression Recognition Based on Deep Evolutional Spatial-Temporal Networks

Learning Expressionlets on Spatio-temporal Manifold for Dynamic Facial Expression Recognition

Posed and Spontaneous Expression Recognition Through Modeling Their Spatial Patterns

Dynamic 3D Facial Expression Modeling Using Laplacian Smooth and Multi-Scale Mesh Matching

Learning facial expression and body gesture visual information for video emotion recognition

High Resolution Acquisition, Learning And Transfer Of Dynamic 3-D Facial Expressions

Muscular Movement Model Based Automatic 3D Facial Expression Recognition.

Muscular Movement Model-Based Automatic 3D/4D Facial Expression Recognition