Abstract:Natural eye movements have primarily been studied for over-learned activities such as tea-making, sandwich-making, and hand-washing, which have a fixed sequence of associated actions. These studies demonstrate a sequential activation of low-level cognitive schemas facilitating task completion. However, whether these action schemas are activated in the same pattern when a task is novel and a sequence of actions must be planned in the moment is unclear. Here, we recorded gaze and body movements in a naturalistic task to study action-oriented gaze behavior. In a virtual environment, subjects moved objects on a life-size shelf to achieve a given order. To compel cognitive planning, we added complexity to the sorting tasks. Fixations aligned with the action onset showed gaze as tightly coupled with the action sequence, and task complexity moderately affected the proportion of fixations on the task-relevant regions. Our analysis revealed that gaze fixations were allocated to action-relevant targets just in time. Planning behavior predominantly corresponded to a greater visual search for task-relevant objects before the action onset. The results support the idea that natural behavior relies on the frugal use of working memory, and humans refrain from encoding objects in the environment to plan long-term actions. Instead, they prefer just-in-time planning by searching for action-relevant items at the moment, directing their body and hand to it, monitoring the action until it is terminated, and moving on to the following action. Eye movements in the natural environment have primarily been studied for over-learned and habitual everyday activities (tea-making, sandwich-making, hand-washing) with a fixed sequence of associated actions. These studies show eye movements correspond to a specific order of actions learned over time. In this study, we were interested in how humans plan and execute actions for tasks that do not have an inherent action sequence. To that end, we asked subjects to sort objects based on object features on a life-size shelf in a virtual environment as we recorded their eye and body movements. We investigated the general characteristics of gaze behavior while acting under natural conditions. Our paper provides a comprehensive approach to preprocess naturalistic gaze data in virtual reality. Furthermore, we provide a data-driven method of analyzing the different action-oriented functions of gaze. The results show that bereft of a predefined action sequence, humans prefer to plan only their immediate actions, where eye movements are used to search for the target object to immediately act on, then to guide the hand towards it and monitor the action until it is terminated. Such a simplistic approach ensures that humans choose sub-optimal behavior over planning under sustained cognitive load.

Just-in-time: Gaze guidance in natural behavior

Just-in-time: gaze guidance in natural behavior

Eye Movements Reveal Spatiotemporal Dynamics of Visually-Informed Planning in Navigation

Oculomotor behavior and perceptual strategies in complex tasks

The Shortest Route Is Not Always the Fastest: Probability-Modeled Stereoscopic Eye Movement Completion Time in VR

The where, what and when of gaze allocation in the lab and the natural environment

Gaze facilitates responsivity during hand coordinated joint attention

Quantifying Gaze Behavior During Real-World Interactions Using Automated Object, Face, and Fixation Detection

Active inhibition and memory promote exploration and search of natural scenes

Oculomotor routines for perceptual judgments

The Roles of Vision and Eye Movements in the Control of Activities of Daily Living

Reactive Gaze during Locomotion in Natural Environments

Intentionally vs. Spontaneously Prolonged Gaze: A MEG Study of Active Gaze-Based Interaction

Low-Dimensional Representations of Visuomotor Coordination for Natural Behavior

MIDAS: Deep learning human action intention prediction from natural eye movement patterns

Gaze Coordination of Groups in Dynamic Events - A Tool to Facilitate Analyses of Simultaneous Gazes Within a Team

Knowing Where to Look: A Planning-based Architecture to Automate the Gaze Behavior of Social Robots

Adaptive Gaze and Hand Coordination while Manipulating and Monitoring the Environment in Parallel

Knowing where to go: Spatial memory guides eye and body movements in a naturalistic visual search task

Combining EEG and Eye-Tracking in Virtual Reality - Obtaining Fixation-Onset ERPs and ERSPs

Free Moving Gaze-related Electroencephalography in Mobile Virtual Environments