Abstract:Current works on multimodal facial expression recognition typically require paired visible and thermal facial images. Although visible cameras are readily available in our daily life, thermal cameras are expensive and less prevalent. It is costly to collect a large quantity of synchronous visible and thermal facial images. To tackle this paired training data bottleneck, we propose an unpaired multimodal facial expression recognition method, which makes full use of the massive number of unpaired visible and thermal images by utilizing thermal images to construct better image representations and classifiers for visible images during training. Specifically, two deep neural networks are trained from visible and thermal images to learn image representations and expression classifiers for two modalities. Then, an adversarial strategy is adopted to force statistical similarity between the learned visible and thermal representations, and to minimize the distribution mismatch between the predictions of the visible and thermal images. Through adversarial learning, the proposed method leverages thermal images to construct better image representations and classifiers for visible images during training, without the requirement of paired data. A decoder network is built upon the visible hidden features in order to preserve some inherent features of the visible view. We also take the variability of the different images’ transferability into account via adaptive classification loss. During testing, only visible images are required and the visible network is used. Thus, the proposed method is appropriate for real-world scenarios, since thermal imaging is rare in these instances. Experiments on two benchmark multimodal expression databases and three visible facial expression databases demonstrate the superiority of the proposed method compared to state-of-the-art methods.

Self-Attentive Contrastive Learning for Conditioned Periocular and Face Biometrics

Periocular embedding learning with consistent knowledge distillation from face

Perturbed Attention-Assisted Siamese Network for Cross-Spectral Periocular Recognition

One shot learning approach for cross spectrum periocular verification

Toward High Quality Facial Representation Learning

Cross-Sensor Periocular Biometrics in a Global Pandemic: Comparative Benchmark and Novel Multialgorithmic Approach

A Robust Face and Ear Based Multimodal Biometric System Using Sparse Representation

Periocular Biometrics: A Modality for Unconstrained Scenarios

Unpaired Multimodal Facial Expression Recognition.

Improvised contrastive loss for improved face recognition in open-set nature

Approximate Multidimensional Discrete Cosine Transform-Based Deep Attention Network for Cross-Spectral Periocular Recognition

Self-Supervised Visual Representations Learning by Contrastive Mask Prediction

Coupled adversarial learning for semi-supervised heterogeneous face recognition

UCoL: Unsupervised Learning of Discriminative Facial Representations via Uncertainty-Aware Contrast

A Deep Feature Fusion Network Based on Multiple Attention Mechanisms for Joint Iris-Periocular Biometric Recognition

MC-LCR: Multimodal contrastive classification by locally correlated representations for effective face forgery detection

Periocular biometrics: databases, algorithms and directions

Geometric Correspondence-Based Multimodal Learning for Ophthalmic Image Analysis

High Boost 3-D Attention Network for Cross-Spectral Periocular Recognition

Biometric identity recognition based on contrastive positive-unlabeled learning

Adversarial Training for Face Recognition Systems using Contrastive Adversarial Learning and Triplet Loss Fine-tuning