Abstract:Objective: To date the vast majority of research in the visual neurosciences have been forced to adopt a highly constrained perspective of the vision system in which stimuli are processed in an open-loop reactive fashion (i.e., abrupt stimulus presentation followed by an evoked neural response). While such constraints enable high construct validity for neuroscientific investigation, the primary outcomes have been a reductionistic approach to isolate the component processes of visual perception. In electrophysiology, of the many neural processes studied under this rubric, the most well-known is, arguably, the P300 evoked response. There is, however, relatively little known about the real-world corollary of this component in free-viewing paradigms where visual stimuli are connected to neural function in a closed-loop. While growing evidence suggests that neural activity analogous to the P300 does occur in such paradigms, it is an open question when this response occurs and what behavioral or environmental factors could be used to isolate this component. Approach: The current work uses convolutional networks to decode neural signals during a free-viewing visual search task in a closed-loop paradigm within an open-world virtual environment. From the decoded activity we construct fixation-locked response profiles that enable estimations of the variable latency of any P300 analogue around the moment of fixation. We then use these estimates to investigate which factors best reduce variable latency and, thus, predict the onset time of the response. We consider measurable, search-related factors encompassing top-down (i.e., goal driven) and bottom-up (i.e., stimulus driven) processes, such as fixation duration and salience. We also consider saccade size as an intermediate factor reflecting the integration of these two systems. Main results: The results show that of these factors only saccade size reliably determines the onset time of P300 analogous activity for this task. Specifically, we find that for large saccades the variability in response onset is small enough to enable analysis using traditional ensemble averaging methods. Significance: The results show that P300 analogous activity does occur during closed-loop, free-viewing visual search while highlighting distinct differences between the open-loop version of this response and its real-world analogue. The results also further establish saccades, and saccade size, as a key factor in real-world visual processing.

Investigating saccade-onset locked EEG signatures of face perception during free-viewing in a naturalistic virtual environment

Combining EEG and Eye-Tracking in Virtual Reality - Obtaining Fixation-Onset ERPs and ERSPs

Electrophysiological correlates of face and object perception: A comparative analysis of 2D laboratory and virtual reality conditions

Free Moving Gaze-related Electroencephalography in Mobile Virtual Environments

(Micro)saccade-related potentials during face recognition: A study combining EEG, eye-tracking, and deconvolution modeling

A comparative analysis of face and object perception in 2D laboratory and virtual reality settings: insights from induced oscillatory responses

Active search signatures in a free-viewing task exploiting concurrent EEG and eye movements recordings

Exploring the Effects of Head Movements and Accompanying Gaze Fixation Switch on Steady-State Visual Evoked Potential

Saccade size predicts onset time of object processing during visual search of an open world virtual environment

A method for synchronized use of EEG and eye tracking in fully immersive VR

Investigating brain mechanisms underlying natural reading by co-registering eye tracking with combined EEG and MEG

The impact of virtual reality and distractors on attentional processes: insights from EEG

A High-Density EEG Study Investigating VR Film Editing and Cognitive Event Segmentation Theory

Saccade onset, not fixation onset, best explains early responses across the human visual cortex during naturalistic vision

Stronger reactivity to social gaze in virtual reality compared to a classical laboratory environment

A comparative experimental study of visual brain event-related potentials to a working memory task: virtual reality head-mounted display versus a desktop computer screen

Real brains in virtual worlds: Validating a novel oddball paradigm in virtual reality

A Test Setting to Compare Spatial Awareness on Paper and in Virtual Reality Using EEG Signals

Comparing encoding mechanisms in realistic virtual reality and conventional 2D laboratory settings: Event-related potentials in a repetition suppression paradigm

A Novel Method to Understand Neural Oscillations During Full-Body Reaching: A Combined EEG and 3D Virtual Reality Study

One-year-later spontaneous EEG features predict visual exploratory human phenotypes