Abstract:Current works on multimodal facial expression recognition typically require paired visible and thermal facial images. Although visible cameras are readily available in our daily life, thermal cameras are expensive and less prevalent. It is costly to collect a large quantity of synchronous visible and thermal facial images. To tackle this paired training data bottleneck, we propose an unpaired multimodal facial expression recognition method, which makes full use of the massive number of unpaired visible and thermal images by utilizing thermal images to construct better image representations and classifiers for visible images during training. Specifically, two deep neural networks are trained from visible and thermal images to learn image representations and expression classifiers for two modalities. Then, an adversarial strategy is adopted to force statistical similarity between the learned visible and thermal representations, and to minimize the distribution mismatch between the predictions of the visible and thermal images. Through adversarial learning, the proposed method leverages thermal images to construct better image representations and classifiers for visible images during training, without the requirement of paired data. A decoder network is built upon the visible hidden features in order to preserve some inherent features of the visible view. We also take the variability of the different images’ transferability into account via adaptive classification loss. During testing, only visible images are required and the visible network is used. Thus, the proposed method is appropriate for real-world scenarios, since thermal imaging is rare in these instances. Experiments on two benchmark multimodal expression databases and three visible facial expression databases demonstrate the superiority of the proposed method compared to state-of-the-art methods.

Multi-Instance Hidden Markov Model For Facial Expression Recognition

Early Facial Expression Recognition Using Hidden Markov Models

Real-Time facial expression recognition system based on HMM and feature point localization

Early Expression Detection Via Online Multi-Instance Learning with Nonlinear Extension

Discriminative Deep Multi-Task Learning for Facial Expression Recognition.

Video-Based Facial Expression Recognition Using Histogram Sequence of Local Gabor Binary Patterns from Three Orthogonal Planes

Facial Expression Recognition Using Multi-Svm with Eigenexpress

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Research on Dynamic Facial Expressions Recognition

Multiple instance learning for labeling faces in broadcasting news video

Expression recognition from video using a coupled hidden Markov model

Multi-stage and multi-branch network with similar expressions label distribution learning for facial expression recognition

Twin attention based multi-task convolutional bidirectional long short term memory for facial expression recognition

Multi-Task Multi-Modal Self-Supervised Learning for Facial Expression Recognition

Human Detection Method Based on Multi-Part Detector and Multi-Instance Learning

A Multi-Task Model for Simultaneous Face Identification and Facial Expression Recognition.

Adaptive key-frame selection-based facial expression recognition via multi-cue dynamic features hybrid fusion

A method for recognizing facial expression intensity based on facial muscle variations

A multi-scale multi-attention network for dynamic facial expression recognition

Unpaired Multimodal Facial Expression Recognition.

Self-supervised Exclusive-Inclusive Interactive Learning for Multi-label Facial Expression Recognition in the Wild