Abstract:Multimodal emotion recognition techniques are increasingly essential for assessing mental states. Image-based methods, however, tend to focus predominantly on overt visual cues and often overlook subtler mental state changes. Psychophysiological research has demonstrated that heart rate (HR) and skin temperature are effective in detecting autonomic nervous system (ANS) activities, thereby revealing these subtle changes. However, traditional HR tools are generally more costly and less portable, while skin temperature analysis usually necessitates extensive manual processing. Advances in remote photoplethysmography (r-PPG) and automatic thermal region of interest (ROI) detection algorithms have been developed to address these issues, yet their accuracy in practical applications remains limited. This study aims to bridge this gap by integrating r-PPG with thermal imaging to enhance prediction performance. Ninety participants completed a 20-min questionnaire to induce cognitive stress, followed by watching a film aimed at eliciting moral elevation. The results demonstrate that the combination of r-PPG and thermal imaging effectively detects emotional shifts. Using r-PPG alone, the prediction accuracy was 77% for cognitive stress and 61% for moral elevation, as determined by a support vector machine (SVM). Thermal imaging alone achieved 79% accuracy for cognitive stress and 78% for moral elevation, utilizing a random forest (RF) algorithm. An early fusion strategy of these modalities significantly improved accuracies, achieving 87% for cognitive stress and 83% for moral elevation using RF. Further analysis, which utilized statistical metrics and explainable machine learning methods including SHapley Additive exPlanations (SHAP), highlighted key features and clarified the relationship between cardiac responses and facial temperature variations. Notably, it was observed that cardiovascular features derived from r-PPG models had a more pronounced influence in data fusion, despite thermal imaging’s higher predictive accuracy in unimodal analysis.

"Sadness smile" curve: Processing emotional information from social network for evaluating thermal comfort perception

Assessing Emotional Responses to the Spatial Quality of Urban Green Spaces Through Self-Report and Face Recognition Measures.

Dynamic Facial Features in Positive-Emotional Speech for Identification of Depressive Tendencies

A Frown-Based Thermal Comfort Detection Method of Facial Emotion Recognition

Discrete Emotions Discovered by Contactless Measurement of Facial Blood Flows

Emotional Perceptions of Thermal Comfort for People Exposed to Green Spaces Characterized Using Streetscapes in Urban Parks

Non-contact Emotion Analysis Based on Videos.

From Faces to Traffic Lights: A Multi-Scale Approach for Emotional State Representation

Extracting human emotions at different places based on facial expressions and spatial clustering analysis

Optimizing Urban Forest Landscape for Better Perceptions of Positive Emotions

Your blush gives you away: detecting hidden mental states with remote photoplethysmography and thermal imaging

Happy-sad expression recognition using emotion geometry feature and support vector machine

Recognition and prediction of elderly thermal sensation based on outdoor facial skin temperature

EmoSense: Computational Intelligence Driven Emotion Sensing Via Wireless Channel Data

Human emotion and cognition recognition from body language of the head using soft computing techniques

Microclimates Potentially Shape Spatial Distribution of Facial Expressions for Urban Forest Visitors: A Regional Study of 30 Parks in North China

Predicting students' thermal sensation votes in university libraries taking into account their mood states

Real-time surveillance-video-based personalized thermal comfort recognition

How Do You Smile? Towards a Comprehensive Smile Analysis System

New Impression of Beijing Hutongs: A Microscale Urban Emotion Measurement Method