Abstract:Micro-expressions, which are spontaneous and difficult to suppress, reveal a person's true emotions. They are characterized by short duration and low intensity, making the task of micro-expression recognition challenging in the field of emotion computing. In recent years, deep learning-based feature extraction and fusion techniques have been widely used for micro-expression recognition, particularly methods based on Vision Transformer that have gained popularity. However, the Vision Transformer-based architecture used in micro-expression recognition involves a significant amount of invalid computation. Additionally, in the traditional two-stream architecture, although separate streams are combined through late fusion, only the output features from the deepest level of the network are utilized for classification, thus limiting the network's ability to capture subtle details due to the lack of fine-grained information. To address these issues, we propose a new two-level spatio-temporal feature fused with a two-stream architecture. This architecture includes a spatial encoder (modified ResNet) for learning texture features of the face, a temporal encoder (Swin Transformer) for learning facial muscle motor features, a feature fusion algorithm for integrating multi-level spatio-temporal features, a classification head, and a weighted average operator for temporal aggregation. The two-stream architecture has the advantage of extracting richer features compared to the single-stream architecture, leading to improved performance. The shifted window scheme of Swin Transformer restricts self-attention computation to non-overlapping local windows and allows cross-window connections, significantly improving the performance and reducing the computation compared to Vision Transformer. Moreover, the modified ResNet is computationally less intensive. Our proposed feature fusion algorithm leverages the similarity in output feature shapes at each stage of the two streams, enabling the effective fusion of multi-level spatio-temporal features. This algorithm results in an improvement of approximately 4% in both the F1 score and the UAR. Comprehensive evaluations conducted on three widely used spontaneous micro-expression datasets (SMIC-HS, CASME II, and SAMM) consistently demonstrate the superiority of our approach over comparative methods. Notably, our approach achieves a UAR exceeding 0.905 on CASME II, making it one of the few frameworks in the published micro-expression recognition literature to achieve such high performance.

LCBP-STGCN: A local cube binary pattern spatial temporal graph convolutional network for micro-expression recognition

Spatiotemporal Convolutional Neural Network with Convolutional Block Attention Module for Micro-Expression Recognition

Sparse Spatiotemporal Descriptor for Micro-Expression Recognition Using Enhanced Local Cube Binary Pattern

Adaptive Temporal Motion Guided Graph Convolution Network for Micro-expression Recognition

Facial Micro-Expression Recognition Based on Multi-Scale Temporal and Spatial Features

Spatio-temporal Convolutional Features with Nested LSTM for Facial Expression Recognition

Facial Micro-Expression Recognition Based on Deep Local-Holistic Network

Spontaneous Facial Micro-Expression Analysis Using Spatiotemporal Completed Local Quantized Patterns

Spatiotemporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-expressions

Discriminative Spatiotemporal Local Binary Pattern with Revisited Integral Projection for Spontaneous Facial Micro-Expression Recognition

Transferring Dual Stochastic Graph Convolutional Network for Facial Micro-expression Recognition

Speech Emotion Recognition Based on Temporal-Spatial Learnable Graph Convolutional Neural Network

Two-Level Spatio-Temporal Feature Fused Two-Stream Network for Micro-Expression Recognition

Micro-expression recognition based on multi-scale 3D residual convolutional neural network

JGULF: Joint global and unilateral local feature network for micro-expression recognition

Multi-Scale Spatio-Temporal Graph Convolutional Network for Facial Expression Spotting

Facial micro-expression recognition using stochastic graph convolutional network and dual transferred learning

STA-GCN - Spatio-Temporal AU Graph Convolution Network for Facial Micro-expression Recognition.

Sparse Tensor Canonical Correlation Analysis for Micro-Expression Recognition

Spontaneous Facial Micro-Expression Recognition using Discriminative Spatiotemporal Local Binary Pattern with an Improved Integral Projection

Micro-expression recognition based on local binary patterns from three orthogonal planes and nearest neighbor method