Abstract:Facial expression is highly correlated with the facial motion. According to whether the temporal information of facial motion is used or not, the facial expression features can be classified as static and dynamic features. The former, which mainly includes the geometric features and appearance features, can be extracted by convolution or other learning filters; the latter, which are aimed to model the dynamic properties of facial motion, can be calculated through optical flow or other methods, respectively. When 3D convolutional neural networks (CNNs) are introduced, the extraction of two different types of features mentioned above becomes easy. In this paper, one 3D CNN architecture is presented to learn the static and dynamic features from facial image sequences and extract high-level dynamic features from optical flow sequences. Two types of dense optical flow, which contain the tracking information of facial muscle movement, are calculated according to different image pair construction methods. One is the common optical flow, and the other is an enhanced optical flow which is called accumulative optical flow. Four components of each type of optical flow are used in experiments. Three databases, two acted databases and one nearly realistic database, are selected to conduct the experiments. The experiments on the two acted databases achieve state-of-the-art accuracy, and indicate that the vertical component of optical flow has an advantage over other components in recognizing facial expression. The experimental results on the three selected databases show that more discriminative features can be learned from image sequences than from optical flow or accumulative optical flow sequences, and the accumulative optical flow contains more motion information than optical flow if the frame distance of the image pairs used to calculate them is not too large.

Learning Deep Facial Expression Features from Image and Optical Flow Sequences Using 3D CNN

Facial Expression Recognition in Video Using 3D-CNN Deep Features Discrimination

Automatic facial expression recognition on a single 3D face by exploring shape deformation.

Deep Networks for Image Motion Estimation

Dynamic Spatio-Temporal Feature Learning via Graph Convolution in 3D Convolutional Networks

Automatic 4D Facial Expression Recognition via Collaborative Cross-domain Dynamic Image Network.

DeepFaceFlow: In-the-wild Dense 3D Facial Motion Estimation

FacialFlowNet: Advancing Facial Optical Flow Estimation with a Diverse Dataset and a Decomposed Model

Deep Networks for Image-to-Image Prediction

Landmarks-assisted Collaborative Deep Framework for Automatic 4D Facial Expression Recognition.

Expression-Guided Deep Joint Learning for Facial Expression Recognition

Classification Mechanism of Convolutional Neural Network for Facial Expression Recognition.

Facial Expression Recognition Using Enhanced Deep 3D Convolutional Neural Networks

Patch Attention Layer of Embedding Handcrafted Features in CNN for Facial Expression Recognition

Learning Dynamics for Video Facial Expression Recognition

Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?

Deep Energies for Estimating Three-Dimensional Facial Pose and Expression

Expression dynamic capture and 3D animation generation method based on deep learning

LBVCNN: Local Binary Volume Convolutional Neural Network for Facial Expression Recognition from Image Sequences

An Efficient Channel Attention CNN for Facial Expression Recognition

A deep-learning-based facial expression recognition method using textural features