Abstract:Facial expression recognition (FER) is embedded in many real-world human-computer interaction tasks, such as online learning, depression recognition and remote diagnosis. However, FER is often hindered by privacy concerns and low recognition accuracy due to inadequate data transfer restrictions on public clouds, insufficient quantities of effective labeled samples and class imbalance. To address the above challenges, we have developed an automatic privacy-preserving learning state recognition system for supervising the quality of online teaching with the cooperation of edge servers and cloud servers to reduce the risk of privacy exposure. In particular, we propose few-shot facial expression recognition with a self-supervised vision transformer (SSF-ViT) by integrating self-supervised learning (SSL) and few-shot learning (FSL) to train a deep learning model with fewer labeled samples. Specifically, a vision transformer (ViT) is jointly pretrained with four self-supervised pretext tasks, including image denoising and reconstruction, image rotation prediction, jigsaw puzzle and masked patch prediction, to obtain a pretrained ViT encoder. Then, the pretrained ViT encoder is used on a lab-controlled labeled FER dataset to extract the spatiotemporal features and implement the FER task to fine-tune the parameters. Finally, we construct a prototype to verify the few-shot classification method for specific expression recognition. Support and query sets are divided in the wild FER dataset, and few-shot classification episodes are constructed. The fine-tuned ViT encoder is used as the feature extractor to build the prototype for each support set category, and the expression classification results are obtained by computing the Euclidean distance between the query samples and the prototypes. The extensive experimental results show that SSF-ViT can achieve recognition accuracies of 74.95%, 66.04%, 63.69% and 90.98% on the FER2013, AffectNet, SFEW 2.0 and RAF-DB datasets, respectively. In addition, SSF-ViT can improve the recognition performance of specific expression categories on these datasets.

Self-supervised facial expression recognition with fine-grained feature selection

DR-FER: Discriminative and Robust Representation Learning for Facial Expression Recognition

Efficient Facial Expression Recognition with Representation Reinforcement Network and Transfer Self-Training for Human–Machine Interaction

Facial expression recognition through multi-level features extraction and fusion

Bridging the Gaps: Utilizing Unlabeled Face Recognition Datasets to Boost Semi-Supervised Facial Expression Recognition

Facial Expression Recognition Based on Zero-Addition Pretext Training and Feature Conjunction-Selection Network in Human–Robot Interaction

A Student Facial Expression Recognition Model Based on Multi-Scale and Deep Fine-Grained Feature Attention Enhancement

Learning informative and discriminative semantic features for robust facial expression recognition

Self-supervised Exclusive-Inclusive Interactive Learning for Multi-label Facial Expression Recognition in the Wild

Facial Expression Recognition with Contrastive Learning and Uncertainty-Guided Relabeling

Feature Super-Resolution Based Facial Expression Recognition for Multi-scale Low-Resolution Faces

Facial Expression Recognition Using Hybrid Features of Pixel and Geometry

Self-supervised vision transformer-based few-shot learning for facial expression recognition

Facial Expression Recognition with Visual Transformers and Attentional Selective Fusion

Facial Expression Recognition Based on Fine-Tuned Channel–Spatial Attention Transformer

Enhanced Dual-Level Representations for Facial Expression Recognition

SAANet: Siamese Action-Units Attention Network for Improving Dynamic Facial Expression Recognition

Joint spatial and scale attention network for multi-view facial expression recognition

A discriminative multiscale feature extraction network for facial expression recognition in the wild

A feature boosted deep learning method for automatic facial expression recognition