Abstract:Virtual reality (VR) is not a new technology but has been in development for decades, driven by advances in computer technology. Currently, VR technology is increasingly being used in applications to enable immersive, yet controlled research settings. Education and entertainment are two important application areas, where VR has been considered a key enabler of immersive experiences and their further advancement. At the same time, the study of human behavior in such innovative environments is expected to contribute to a better design of VR applications. Therefore, modern VR devices are consistently equipped with eye-tracking technology, enabling thus further studies of human behavior through the collection of process data. In particular, eye-tracking technology in combination with machine learning techniques and explainable models can provide new insights for a deeper understanding of human behavior during immersion in virtual environments. In this work, a systematic computational framework based on eye-tracking and behavioral user data and state-of-the-art machine learning approaches is proposed to understand human behavior and individual differences in VR contexts. This computational framework is then employed in three user studies across two different domains. In the educational domain, two different immersive VR classrooms were created where students can learn and teachers can train. In terms of VR entertainment, eye movements open a new avenue to evaluate VR locomotion techniques from the perspective of user cognitive load and user experience. This work paves the way for assessing human behavior in VR scenarios and provides profound insights into the way of designing, evaluating, and improving interactive VR systems. In particular, more effective and customizable virtual environments can be created to provide users with tailored experiences.

Inferring Human Vision in a Human-Like Way: Key Factors Influencing the Cognitive Processing of Level-1 Visual Perspective-Taking

People Do Not Automatically Take the Level-1 Visual Perspective of Humanoid Robot Avatars

Better Than Humans: a Method for Inferring Consumer Shopping Intentions by Reading Facial Expressions

When Computer Vision Gazes at Cognition

Vision Language Models See What You Want but not What You See

Mentalizing Another's Visual World—A Novel Exploration Via Motion Aftereffect

I prefer what you can see: The role of visual perspective-taking on the gaze-liking effect

Measuring and modeling the perception of natural and unconstrained gaze in humans and machines

Manipulating avatar age and gender in level-2 visual perspective taking

Comparing Visual Reasoning in Humans and AI

Visual perspective taking for avatars in a Simon task

An Adaptive Vision System Toward Implicit Human Computer Interaction

Human-like Visual Learning and Reasoning

Towards human-like and transhuman perception in AI 2.0: a review

Towards Human-Leveled Vision Systems

Visual Perspective Taking for Opponent Behavior Modeling

An Information Processing Pattern from Robotics Predicts Unknown Properties of the Human Visual System

Eyes on the Game: Deciphering Implicit Human Signals to Infer Human Proficiency, Trust, and Intent

Virtual Reflections on a Dynamic 2D Eye Model Improve Spatial Reference Identification

Visual perspective taking and action understanding

Assessment of Human Behavior in Virtual Reality by Eye Tracking