Abstract:Current works on multimodal facial expression recognition typically require paired visible and thermal facial images. Although visible cameras are readily available in our daily life, thermal cameras are expensive and less prevalent. It is costly to collect a large quantity of synchronous visible and thermal facial images. To tackle this paired training data bottleneck, we propose an unpaired multimodal facial expression recognition method, which makes full use of the massive number of unpaired visible and thermal images by utilizing thermal images to construct better image representations and classifiers for visible images during training. Specifically, two deep neural networks are trained from visible and thermal images to learn image representations and expression classifiers for two modalities. Then, an adversarial strategy is adopted to force statistical similarity between the learned visible and thermal representations, and to minimize the distribution mismatch between the predictions of the visible and thermal images. Through adversarial learning, the proposed method leverages thermal images to construct better image representations and classifiers for visible images during training, without the requirement of paired data. A decoder network is built upon the visible hidden features in order to preserve some inherent features of the visible view. We also take the variability of the different images’ transferability into account via adaptive classification loss. During testing, only visible images are required and the visible network is used. Thus, the proposed method is appropriate for real-world scenarios, since thermal imaging is rare in these instances. Experiments on two benchmark multimodal expression databases and three visible facial expression databases demonstrate the superiority of the proposed method compared to state-of-the-art methods.

Electromagnetic Imaging Boosted Visual Object Recognition under Difficult Visual Conditions

Advancing infrared and visible image fusion with an enhanced multiscale encoder and attention-based networks

CMEFusion: Cross-Modal Enhancement and Fusion of FIR and Visible Images

Unpaired Multimodal Facial Expression Recognition.

MEEAFusion: Multi-Scale Edge Enhancement and Joint Attention Mechanism Based Infrared and Visible Image Fusion

VAM-Net: Vegetation-Attentive Deep Network for Multi-modal Fusion of Visible-Light and Vegetation-Sensitive Images

Target Recognition Based on Infrared and Visible Image Fusion and Improved YOLOv8 Algorithm

CMFA_Net: A cross-modal feature aggregation network for infrared-visible image fusion

Unifying Visual and Semantic Feature Spaces with Diffusion Models for Enhanced Cross-Modal Alignment

Object Recognition via Adaptive Multi-level Feature Integration

Cross-Modality Attentive Feature Fusion for Object Detection in Multispectral Remote Sensing Imagery

MAVEN: A Memory Augmented Recurrent Approach for Multimodal Fusion

Elevation Information-Guided Multimodal Fusion Robust Framework for Remote Sensing Image Segmentation

FusionMamba: Dynamic Feature Enhancement for Multimodal Image Fusion with Mamba

MVSFusion: infrared and visible image fusion method for multiple visual scenarios

Equivariant Multi-Modality Image Fusion

MVF: A Novel Infrared and Visible Image Fusion Approach Based on the Morphing Convolutional Structure and the Light-Weight Visual State Space Block

ACDF-YOLO: Attentive and Cross-Differential Fusion Network for Multimodal Remote Sensing Object Detection

Boosting Multi-modal Ocular Recognition via Spatial Feature Reconstruction and Unsupervised Image Quality Estimation

Adaptive low light visual enhancement and high-significant target detection for infrared and visible image fusion

MSCS: Multi-stage Feature Learning with Channel-Spatial Attention Mechanism for Infrared and Visible Image Fusion