Abstract:We introduce a novel automatic detection method for facial action units (AUs) that leverages both spatial and temporal data, enhancing accuracy and robustness in expression analysis and facial animation. Our approach utilizes a Temporal feature Combination and Feature Reassignment (TC&FR) module to transform and fuse features across multiple subjects and temporal sequences. Moreover, by integrating a Regional Attention (RA) encoder and a transformer model, our method refines the extraction and processing of regional features, ensuring more precise identification and analysis of AUs. This integration not only harnesses identity‐independent features but also maximizes the temporal context, significantly improving the reliability of AU predictions. Facial action units (AUs) encode the activations of facial muscle groups, playing a crucial role in expression analysis and facial animation. However, current deep learning AU detection methods primarily focus on single‐image analysis, which limits the exploitation of rich temporal context for robust outcomes. Moreover, the scale of available datasets remains limited, leading models trained on these datasets to tend to suffer from overfitting issues. This paper proposes a novel AU detection method integrating spatial and temporal data with inter‐subject feature reassignment for accurate and robust AU predictions. Our method first extracts regional features from facial images. Then, to effectively capture both the temporal context and identity‐independent features, we introduce a temporal feature combination and feature reassignment (TC&FR) module, which transforms single‐image features into a cohesive temporal sequence and fuses features across multiple subjects. This transformation encourages the model to utilize identity‐independent features and temporal context, thus ensuring robust prediction outcomes. Experimental results demonstrate the enhancements brought by the proposed modules and the state‐of‐the‐art (SOTA) results achieved by our method.

CCT: A Cross-Concat and Temporal Neural Network for Multi-Label Action Unit Detection

Region And Temporal Dependency Fusion For Multi-Label Action Unit Detection

A Channel-Wise Spatial-Temporal Aggregation Network for Action Recognition

Facial Action Units Detection Aided by Global-Local Expression Embedding

Facial Action Unit Detection by Exploring the Weak Relationships Between AU Labels

Facial action units detection using temporal context and feature reassignment

AU R-CNN: Encoding Expert Prior Knowledge into R-CNN for Action Unit Detection

FACS3D-Net: 3D Convolution Based Spatiotemporal Representation for Action Unit Detection

An Action Unit Co-Occurrence Constraint 3DCNN Based Action Unit Recognition Approach.

AQ-CSL: Attention Querying Facial Action Unit Detection Net Withcross Subject Learning

Multi-Scale Region with Local Relationship Learning for Facial Action Unit Detection

Facial Action Unit Detection Via Adaptive Attention and Relation.

Facial Action Unit Detection Using Attention and Relation Learning

Action Unit Detection with Joint Adaptive Attention and Graph Relation

Learning Contrastive Feature Representations for Facial Action Unit Detection

Joint Patch And Multi-Label Learning For Facial Action Unit Detection

Heterogeneous Spatio-Temporal Relation Learning Network for Facial Action Unit Detection.

MMA-Net: Multi-view Mixed Attention Mechanism for Facial Action Unit Detection

Facial Action Unit Recognition Based on Self-Attention Spatiotemporal Fusion

Facial Action Unit Detection with Multilayer Fused Multi-Task and Multi-Label Deep Learning Network

Action Unit Detection by Exploiting Spatial-Temporal and Label-Wise Attention with Transformer