Abstract:Multi-modal signals have become essential data for emotion recognition since they can represent emotions more comprehensively. However, in real-world environments, it is often impossible to acquire complete data on multi-modal signals, and the problem of missing modalities causes severe performance degradation in emotion recognition. Therefore, this paper represents the first attempt to use a transformer-based architecture, aiming to fill the modality-incomplete data from partially observed data for multi-modal emotion recognition (MER). Concretely, this paper proposes a novel unified model called transformer autoencoder (TAE), comprising a modality-specific hybrid transformer encoder, an inter-modality transformer encoder, and a convolutional decoder. The modality-specific hybrid transformer encoder bridges a convolutional encoder and a transformer encoder, allowing the encoder to learn local and global context information within each particular modality. The inter-modality transformer encoder builds and aligns global cross-modal correlations and models long-range contextual information with different modalities. The convolutional decoder decodes the encoding features to produce more precise recognition. Besides, a regularization term is introduced into the convolutional decoder to force the decoder to fully leverage the complete and incomplete data for emotional recognition of missing data. 96.33%, 95.64%, and 92.69% accuracies are attained on the available data of the DEAP and SEED-IV datasets, and 93.25%, 92.23%, and 81.76% accuracies are obtained on the missing data. Particularly, the model acquires a 5.61% advantage with 70% missing data, demonstrating that the model outperforms some state-of-the-art approaches in incomplete multi-modal learning.

Estimating Missing Tensor Data by Face Synthesis for Expression Recognition

Video Tracked Facial Expression Animation

Automatic facial expression recognition on a single 3D face by exploring shape deformation.

A Data-Driven Approach for Facial Expression Retargeting in Video

Sparse and Low-rank Tucker Decomposition with Its Application to 2D+3D Facial Expression Recognition

A data-driven approach for facial expression synthesis in video

Jointly Optimizing Expressional and Residual Models for 3D Facial Expression Removal.

Real-time Facial Expression Recognition "In The Wild'' by Disentangling 3D Expression from Identity

Transformer-Based Multimodal Emotional Perception for Dynamic Facial Expression Recognition in the Wild

Expression snippet transformer for robust video-based facial expression recognition

Bayesian Tensor Approach for 3-D Face Modeling.

Facial expression editing in video using a temporally-smooth factorization

Incorporating eyebrow and eye state information for facial expression recognition in mask-obscured scenes

Triplet Loss-less Center Loss Sampling Strategies in Facial Expression Recognition Scenarios

A discriminative multiscale feature extraction network for facial expression recognition in the wild

Expression-independent Face Recognition Based on Higher-Order Singular Value Decomposition

Expression recognition from 3D dynamic faces using robust spatio-temporal shape features

Sparse Orthogonal Tucker Decomposition for 2D+3D Facial Expression Recognition

Active AU Based Patch Weighting for Facial Expression Recognition

A novel transformer autoencoder for multi-modal emotion recognition with incomplete data

Identity-aware Facial Expression Recognition via Deep Metric Learning based on Synthesized Images