Abstract:Current works on multimodal facial expression recognition typically require paired visible and thermal facial images. Although visible cameras are readily available in our daily life, thermal cameras are expensive and less prevalent. It is costly to collect a large quantity of synchronous visible and thermal facial images. To tackle this paired training data bottleneck, we propose an unpaired multimodal facial expression recognition method, which makes full use of the massive number of unpaired visible and thermal images by utilizing thermal images to construct better image representations and classifiers for visible images during training. Specifically, two deep neural networks are trained from visible and thermal images to learn image representations and expression classifiers for two modalities. Then, an adversarial strategy is adopted to force statistical similarity between the learned visible and thermal representations, and to minimize the distribution mismatch between the predictions of the visible and thermal images. Through adversarial learning, the proposed method leverages thermal images to construct better image representations and classifiers for visible images during training, without the requirement of paired data. A decoder network is built upon the visible hidden features in order to preserve some inherent features of the visible view. We also take the variability of the different images’ transferability into account via adaptive classification loss. During testing, only visible images are required and the visible network is used. Thus, the proposed method is appropriate for real-world scenarios, since thermal imaging is rare in these instances. Experiments on two benchmark multimodal expression databases and three visible facial expression databases demonstrate the superiority of the proposed method compared to state-of-the-art methods.

Gender recognition from visible and thermal infrared facial images

Modality-transfer Generative Adversarial Network and Dual-Level Unified Latent Representation for Visible Thermal Person Re-Identification

Fusion of visible and thermal images for facial expression recognition

Expression Recognition from Visible Images with the Help of Thermal Images

Spontaneous Facial Expression Recognition by Using Feature-Level Fusion of Visible and Thermal Infrared Images

A Fusion Framework For Face Recognition Under Varying Illumination Based On Multi-Scale Analysis

Thermal Augmented Expression Recognition

High-Resolution Face Fusion for Gender Conversion

Fusing gait and face cues for human gender recognition

Unpaired Multimodal Facial Expression Recognition.

Facial Expression Recognition from Infrared Thermal Images Using Temperature Difference by Voting

Facial Expression Recognition Enhanced by Thermal Images Through Adversarial Learning.

Facial Expression Recognition from Infrared Thermal Videos.

Fuzzy Fusion for Face Recognition

Learning and Fusing Multimodal Features from and for Multi-task Facial Computing

Facial Expression Recognition Using Visible, IR, and MSX Images by Early and Late Fusion of Deep Learning Models

Emotion Recognition from Thermal Infrared Images Using Deep Boltzmann Machine

A Novel Infrared and Visible Image Fusion Approach Based on Adversarial Neural Network

A unified framework for thermal face recognition

Gender Recognition from Human-Body Images Using Visible-Light and Thermal Camera Videos Based on a Convolutional Neural Network for Image Feature Extraction

Enhancing thermal facial recognition leveraging large datasets and hybrid algorithms