Abstract:The cognitive state of a person can be categorized using the circumplex model of emotional states, a continuous model of two dimensions: arousal and valence. The purpose of this research is to select a machine learning model(s) to be integrated into a virtual reality (VR) system that runs cognitive remediation exercises for people with mental health disorders. As such, the prediction of emotional states is essential to customize treatments for those individuals. We exploit the Remote Collaborative and Affective Interactions (RECOLA) database to predict arousal and valence values using machine learning techniques. RECOLA includes audio, video, and physiological recordings of interactions between human participants. To allow learners to focus on the most relevant data, features are extracted from raw data. Such features can be predesigned, learned, or extracted implicitly using deep learners. Our previous work on video recordings focused on predesigned and learned visual features. In this paper, we extend our work onto deep visual features. Our deep visual features are extracted using the MobileNet-v2 convolutional neural network (CNN) that we previously trained on RECOLA's video frames of full/half faces. As the final purpose of our work is to integrate our solution into a practical VR application using head-mounted displays, we experimented with half faces as a proof of concept. The extracted deep features were then used to predict arousal and valence values via optimizable ensemble regression. We also fused the extracted visual features with the predesigned visual features and predicted arousal and valence values using the combined feature set. In an attempt to enhance our prediction performance, we further fused the predictions of the optimizable ensemble model with the predictions of the MobileNet-v2 model. After decision fusion, we achieved a root mean squared error (RMSE) of 0.1140, a Pearson's correlation coefficient (PCC) of 0.8000, and a concordance correlation coefficient (CCC) of 0.7868 on arousal predictions. We achieved an RMSE of 0.0790, a PCC of 0.7904, and a CCC of 0.7645 on valence predictions.

An experimental approach to decoding human reactions through mixed measurements

EMOTIF – A system for modeling 3D environment evaluation based on 7D emotional vectors

Thelxinoë: Recognizing Human Emotions Using Pupillometry and Machine Learning

Facial-Video-Based Physiological Signal Measurement: Recent Advances and Affective Applications

Real vs. immersive-virtual emotional experience: Analysis of psycho-physiological patterns in a free exploration of an art museum

Measuring and Predicting Multisensory Reaction Latency: A Probabilistic Model for Visual-Auditory Integration

Neurophysiological and Subjective Analysis of VR Emotion Induction Paradigm

Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing

Multimodal Physiological Analysis of Cognitive Affective Effect of Immersive Learning in Virtual Reality

Virtual Reality for Emotion Elicitation -- A Review

Immersive Emotion Analysis in VR Environments: A Sensor-Based Approach to Prevent Distortion

Assessment and Application of Digital Museum Visitors' Emotional Experience Based on Virtual Reality Technology and Emotion Recognition Algorithm

Heart rate variability analysis for the assessment of immersive emotional arousal using virtual reality: Comparing real and virtual scenarios

EEG-based Emotion Recognition in Immersive Virtual Reality: Meeting the Requirement of Accuracy and Computational Efficiency

Studying Consumer Emotions and Purchase Preferences in a Virtual Reality Environment: A Bibliometric Analysis

Human Emotions Analysis and Recognition Using EEG Signals in Response to 360$^\circ$ Videos

Emotion Recognition Based ECG in Olfactory-enhanced VR Scenes

Predicting the Arousal and Valence Values of Emotional States Using Learned, Predesigned, and Deep Visual Features

Self-Assessed Experience of Emotional Involvement in Sensory Analysis Performed in Virtual Reality

Towards electrophysiological measurement of presence in virtual reality through auditory oddball stimuli

The eyes have it: Inter-subject correlations of pupillary responses for audience response measurement in VR