Abstract:In the billions of faces that are shaped by thousands of different cultures and ethnicities, one thing remains universal: the way emotions are expressed. To take the next step in human-machine interactions, a machine (e.g., a humanoid robot) must be able to clarify facial emotions. Allowing systems to recognize micro-expressions affords the machine a deeper dive into a person's true feelings, which will take human emotion into account while making optimal decisions. For instance, these machines will be able to detect dangerous situations, alert caregivers to challenges, and provide appropriate responses. Micro-expressions are involuntary and transient facial expressions capable of revealing genuine emotions. We propose a new hybrid neural network (NN) model capable of micro-expression recognition in real-time applications. Several NN models are first compared in this study. Then, a hybrid NN model is created by combining a convolutional neural network (CNN), a recurrent neural network (RNN, e.g., long short-term memory (LSTM)), and a vision transformer. The CNN can extract spatial features (within a neighborhood of an image), whereas the LSTM can summarize temporal features. In addition, a transformer with an attention mechanism can capture sparse spatial relations residing in an image or between frames in a video clip. The inputs of the model are short facial videos, while the outputs are the micro-expressions recognized from the videos. The NN models are trained and tested with publicly available facial micro-expression datasets to recognize different micro-expressions (e.g., happiness, fear, anger, surprise, disgust, sadness). Score fusion and improvement metrics are also presented in our experiments. The results of our proposed models are compared with that of literature-reported methods tested on the same datasets. The proposed hybrid model performs the best, where score fusion can dramatically increase recognition performance.

Expression snippet transformer for robust video-based facial expression recognition

Short and Long Range Relation Based Spatio-Temporal Transformer for Micro-Expression Recognition

Efficient Video Transformers via Spatial-Temporal Token Merging for Action Recognition

Micro-expression Spotting with Multi-scale Local Transformer in Long Videos

VaBTFER: An Effective Variant Binary Transformer for Facial Expression Recognition

CNN-Transformer Architecture Solution for Compound Facial Expression Recognition

Transformer-Based Multimodal Emotional Perception for Dynamic Facial Expression Recognition in the Wild

Robust facial expression recognition with Transformer Block Enhancement Module

Facial Expression Recognition With Visual Transformers and Attentional Selective Fusion

Enhanced Hybrid Vision Transformer with Multi-Scale Feature Integration and Patch Dropping for Facial Expression Recognition

Fine-Grained Temporal-Enhanced Transformer for Dynamic Facial Expression Recognition

A Joint Local Spatial and Global Temporal CNN-Transformer for Dynamic Facial Expression Recognition

Facial Micro-Expression Recognition Enhanced by Score Fusion and a Hybrid Model from Convolutional LSTM and Vision Transformer

Facial Expression Recognition Based on Fine-Tuned Channel–Spatial Attention Transformer

SpotFormer: Multi-Scale Spatio-Temporal Transformer for Facial Expression Spotting

Spatial-Temporal Graphs Plus Transformers for Geometry-Guided Facial Expression Recognition

A Self-Attention Based Method for Facial Expression Recognition

Facial micro-expression recognition using three-stream vision transformer network with sparse sampling and relabeling

Coarse-to-Fine Cascaded Networks with Smooth Predicting for Video Facial Expression Recognition

Attention on Emotions: A Vision Transformer Approach to Advancing Facial Expression Recognition