Abstract:Visible facial images provide geometric and appearance patterns of facial expressions and are sensitive to illumination changes. Thermal facial images record facial temperature distribution and are robust to light conditions. Therefore, expression recognition is enhanced by visible and thermal image fusion. In most cases, only visible images are available due to the widespread popularity of visible cameras and the high cost of thermal cameras. Thus, we propose a novel visible expression recognition method by using thermal infrared (IR) data as privileged information, which is only available during training. Specifically, we first learn a deep model for visible images and thermal images. Then we use the learned feature representations to train support vector machine (SVM) classifiers for expression classification. We jointly refine the deep models as well as the SVM classifiers for both thermal images and visible images by imposing the constraint that the outputs of the SVM classifiers from two views are similar. Thermal IR images during training are then exploited to construct better facial representations and expression classifiers from visible images. We extend the proposed thermal augmented expression recognition method for partially unpaired data, acknowledging that visible images and thermal images maybe not be recorded synchronously. Experimental resulton the MAHNOB laughter database demonstrate that the proposed thermal augmented expression recognition method can effectively exploit thermal IR images’ supplementary role for visible facial expression recognition during training to obtain better facial representations and a better visible expression classifier. The proposed thermal augmented expression recognition method achieves state-of-the-art expression recognition performance for both paired and unpaired facial images.

Spontaneous Facial Expression Recognition by Using Feature-Level Fusion of Visible and Thermal Infrared Images

Fusion of visible and thermal images for facial expression recognition

Expression Recognition from Visible Images with the Help of Thermal Images

Facial Expression Recognition from Infrared Thermal Videos.

Thermal Augmented Expression Recognition

Facial Expression Recognition from Infrared Thermal Images Using Temperature Difference by Voting

A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference

Spatio-temporal Analysis for Infrared Facial Expression Recognition from Videos.

Fusion of Visible and Infrared Imagery for Face Recognition

Facial Expression Recognition Enhanced by Thermal Images Through Adversarial Learning.

Gender recognition from visible and thermal infrared facial images

Posed and spontaneous expression distinguishment from infrared thermal images

Face emotion recognition based on infrared thermal imagery by applying machine learning and parallelism

Infrared Facial Expression Recognition Using Wavelet Transform

Facial Expression Recognition Using Visible, IR, and MSX Images by Early and Late Fusion of Deep Learning Models

Near infrared face recognition using Zernike moments and Hermite kernels

Facial Expression Recognition Using Deep Boltzmann Machine from Thermal Infrared Images

Emotion Recognition from Thermal Infrared Images Using Deep Boltzmann Machine

Deep Spatial-Temporal Feature Fusion for Facial Expression Recognition in Static Images

Facial Expression Recognition With Fusion Features Extracted From Salient Facial Areas

A Spontaneous Facial Expression Recognition Method Using Head Motion and AAM Features