Abstract:Facial expression recognition (FER) is embedded in many real-world human-computer interaction tasks, such as online learning, depression recognition and remote diagnosis. However, FER is often hindered by privacy concerns and low recognition accuracy due to inadequate data transfer restrictions on public clouds, insufficient quantities of effective labeled samples and class imbalance. To address the above challenges, we have developed an automatic privacy-preserving learning state recognition system for supervising the quality of online teaching with the cooperation of edge servers and cloud servers to reduce the risk of privacy exposure. In particular, we propose few-shot facial expression recognition with a self-supervised vision transformer (SSF-ViT) by integrating self-supervised learning (SSL) and few-shot learning (FSL) to train a deep learning model with fewer labeled samples. Specifically, a vision transformer (ViT) is jointly pretrained with four self-supervised pretext tasks, including image denoising and reconstruction, image rotation prediction, jigsaw puzzle and masked patch prediction, to obtain a pretrained ViT encoder. Then, the pretrained ViT encoder is used on a lab-controlled labeled FER dataset to extract the spatiotemporal features and implement the FER task to fine-tune the parameters. Finally, we construct a prototype to verify the few-shot classification method for specific expression recognition. Support and query sets are divided in the wild FER dataset, and few-shot classification episodes are constructed. The fine-tuned ViT encoder is used as the feature extractor to build the prototype for each support set category, and the expression classification results are obtained by computing the Euclidean distance between the query samples and the prototypes. The extensive experimental results show that SSF-ViT can achieve recognition accuracies of 74.95%, 66.04%, 63.69% and 90.98% on the FER2013, AffectNet, SFEW 2.0 and RAF-DB datasets, respectively. In addition, SSF-ViT can improve the recognition performance of specific expression categories on these datasets.

Variance-Aware Bi-Attention Expression Transformer for Open-Set Facial Expression Recognition in the Wild

Facial Expression Recognition With Visual Transformers and Attentional Selective Fusion

Open-Set Facial Expression Recognition

Knowledge Conditioned Variational Learning for One-Class Facial Expression Recognition

VaBTFER: An Effective Variant Binary Transformer for Facial Expression Recognition

Open-Set Video-based Facial Expression Recognition with Human Expression-sensitive Prompting

Facial Expression Recognition Based on Fine-Tuned Channel–Spatial Attention Transformer

Exploring Facial Expression Recognition through Semi-Supervised Pretraining and Temporal Modeling

Hybrid Attention-Aware Learning Network for Facial Expression Recognition in the Wild

Exploring Large-scale Unlabeled Faces to Enhance Facial Expression Recognition

Expression Recognition Analysis in the Wild

Benchmarking Deep Facial Expression Recognition: An Extensive Protocol with Balanced Dataset in the Wild

Self-supervised vision transformer-based few-shot learning for facial expression recognition

Knowledge-Enhanced Facial Expression Recognition with Emotional-to-Neutral Transformation

Attention on Emotions: A Vision Transformer Approach to Advancing Facial Expression Recognition

Multi-relations Aware Network for in-the-wild Facial Expression Recognition

Facial Expression Recognition Methods in the Wild Based on Fusion Feature of Attention Mechanism and LBP

Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition

FER-former: Multi-modal Transformer for Facial Expression Recognition

A Fine-Grained Facial Expression Database for End-to-End Multi-Pose Facial Expression Recognition