Abstract:The cognitive state of a person can be categorized using the circumplex model of emotional states, a continuous model of two dimensions: arousal and valence. The purpose of this research is to select a machine learning model(s) to be integrated into a virtual reality (VR) system that runs cognitive remediation exercises for people with mental health disorders. As such, the prediction of emotional states is essential to customize treatments for those individuals. We exploit the Remote Collaborative and Affective Interactions (RECOLA) database to predict arousal and valence values using machine learning techniques. RECOLA includes audio, video, and physiological recordings of interactions between human participants. To allow learners to focus on the most relevant data, features are extracted from raw data. Such features can be predesigned, learned, or extracted implicitly using deep learners. Our previous work on video recordings focused on predesigned and learned visual features. In this paper, we extend our work onto deep visual features. Our deep visual features are extracted using the MobileNet-v2 convolutional neural network (CNN) that we previously trained on RECOLA's video frames of full/half faces. As the final purpose of our work is to integrate our solution into a practical VR application using head-mounted displays, we experimented with half faces as a proof of concept. The extracted deep features were then used to predict arousal and valence values via optimizable ensemble regression. We also fused the extracted visual features with the predesigned visual features and predicted arousal and valence values using the combined feature set. In an attempt to enhance our prediction performance, we further fused the predictions of the optimizable ensemble model with the predictions of the MobileNet-v2 model. After decision fusion, we achieved a root mean squared error (RMSE) of 0.1140, a Pearson's correlation coefficient (PCC) of 0.8000, and a concordance correlation coefficient (CCC) of 0.7868 on arousal predictions. We achieved an RMSE of 0.0790, a PCC of 0.7904, and a CCC of 0.7645 on valence predictions.

Enabling Intelligent Immersive Learning Using Deep Learning-based Learner Confidence Estimation

Confidence Estimation Using Machine Learning in Immersive Learning Environments

Virtual Reality and 6G Based Smart Classroom Teaching Using Artificial Intelligence

Improved AdaBoost for Virtual Reality Experience Prediction Based on Long Short-Term Memory Network

Predicting the Arousal and Valence Values of Emotional States Using Learned, Predesigned, and Deep Visual Features

Sustainable Educational Metaverse Content and System Based on Deep Learning for Enhancing Learner Immersion

Deep Learning Development Environment in Virtual Reality

AI-Powered VR for Enhanced Learning Compared to Traditional Methods

Measuring the Sense of Presence and Learning Efficacy in Immersive Virtual Assembly Training

Exploring Interactive Learning Environments Based on Augmented Reality Technology

Case report: Quantitative recognition of virtual human technology acceptance based on efficient deep neural network algorithm

Confidence-Aware Learning Assistant

A Machine Learning Approach for Predicting Upper Limb Motion Intentions with Multimodal Data in Virtual Reality

Unlocking the Potential: A Comprehensive Evaluation of Augmented Reality and Virtual Reality in Education

Deep Visual Computing of Behavioral Characteristics in Complex Scenarios and Embedded Object Recognition Applications

Metaversal Learning Environments: Measuring, predicting and improving interpersonal effectiveness

Construction of Virtual Reality-Interactive Classroom Based on Deep Learning Algorithm

Emotion Based Prediction in the Context of Optimized Trajectory Planning for Immersive Learning

Long short-term memory network-based user behavior analysis in virtual reality training system—a case study of the ship communication and navigation equipment training

Scientific and Fantastical: Creating Immersive, Culturally Relevant Learning Experiences with Augmented Reality and Large Language Models

Revolutionizing online education: Advanced facial expression recognition for real-time student progress tracking via deep learning model