Abstract:Previous research on Facial Expression Recognition (FER) assisted by facial landmarks mainly focused on single-task learning or hard-parameter sharing based multi-task learning. However, soft-parameter sharing based methods have not been explored in this area. Therefore, this paper adopts Facial Landmark Detection (FLD) as the auxiliary task and explores new multi-task learning strategies for FER. First, three classical multi-task structures, including Hard-Parameter Sharing (HPS), Cross-Stitch Network (CSN), and Partially Shared Multi-task Convolutional Neural Network (PS-MCNN), are used to verify the advantages of multi-task learning for FER. Then, we propose a new end-to-end Co-attentive Multi-task Convolutional Neural Network (CMCNN), which is composed of the Channel Co-Attention Module (CCAM) and the Spatial Co-Attention Module (SCAM). Functionally, the CCAM generates the channel co-attention scores by capturing the inter-dependencies of different channels between FER and FLD tasks. The SCAM combines the max- and average-pooling operations to formulate the spatial co-attention scores. Finally, we conduct extensive experiments on four widely used benchmark facial expression databases, including RAF, SFEW2, CK+, and Oulu-CASIA. Extensive experimental results show that our approach achieves better performance than single-task and multi-task baselines, fully validating multi-task learning’s effectiveness and generalizability 1 1 Codes and detailed instructions can be available at https://github.com/thuiar/cmcnn. .

Residual multi-task learning for facial landmark localization and expression recognition

Multi-task, multi-label and multi-domain learning with residual convolutional networks for emotion recognition

A New Deep Learning Method for Multi-label Facial Expression Recognition Based on Local Constraint Features

Dual Learning for Joint Facial Landmark Detection and Action Unit Recognition

Co-attentive multi-task convolutional neural network for facial expression recognition

Deep Multi-task Learning for Facial Expression Recognition and Synthesis Based on Selective Feature Sharing

Landmarks-assisted Collaborative Deep Framework for Automatic 4D Facial Expression Recognition.

A Real-Time Multi-Task Learning System for Joint Detection of Face, Facial Landmark and Head Pose

Multi-Task Learning Via SA-FPN and EJ-Head

Micro-Expression Recognition Based on Multi-task Learning and Resnet18

Deep Cross Residual Learning for Multitask Visual Recognition

Deep Multi-Center Learning for Face Alignment

Multi-task Learning of Cascaded CNN for Facial Attribute Classification.

Multi-task Convolution Network for Face Alignment

Discriminative Deep Multi-Task Learning for Facial Expression Recognition.

Learning Deep Representation for Face Alignment with Auxiliary Attributes

Multitask learning and CNN for application of face recognition.

A Multi-Task Model for Simultaneous Face Identification and Facial Expression Recognition.

Cross-Task Multi-Branch Vision Transformer for Facial Expression and Mask Wearing Classification

Collaborative Learning for Language and Speaker Recognition.

Multitask Learning Strategy with Pseudo-Labeling: Face Recognition, Facial Landmark Detection, and Head Pose Estimation