Combining EEG and Eye-Tracking in Virtual Reality - Obtaining Fixation-Onset ERPs and ERSPs

Debora Nolte,Marc Vidal De Palol,Ashima Keshava,John Madrid-Carvajal,Anna L. Gert,Eva-Marie von Butler,Pelin Kömürlüoğlu,Peter König

DOI: https://doi.org/10.1101/2024.04.24.590885

2024-04-28

Abstract:Extensive research conducted in controlled laboratory settings has prompted an inquiry into how results can be generalized to real-world situations influenced by the subjects’ actions. Virtual reality lends itself ideally to investigating complex situations but requires accurate classification of eye movements, especially when combining it with time-sensitive data such as EEG. We recorded eye-tracking data in virtual reality and classified it into gazes and saccades using a velocity-based classification algorithm, and we cut the continuous data into smaller segments to deal with varying noise levels, as introduced in the REMoDNav algorithm. Furthermore, we corrected for participants’ translational movement in virtual reality. Various measures, including visual inspection, event durations, and the velocity and dispersion distributions before and after gaze onset, indicate that we can accurately classify the continuous, free-exploration data. Combining the classified eye-tracking with the EEG data, we generated fixation-onset ERPs and ERSPs, providing further evidence for the quality of the eye movement classification and timing of the onset of events. Finally, investigating the correlation between single trials and the average ERP and ERSP identified that fixation-onset ERSPs are less time-sensitive, require fewer repetitions of the same behavior, and are potentially better suited to study EEG signatures in naturalistic settings. We modified, designed, and tested an algorithm that allows the combination of EEG and eye-tracking data recorded in virtual reality.

Neuroscience

What problem does this paper attempt to address?

The paper attempts to address the issue of how to accurately combine electroencephalogram (EEG) and eye-tracking data in a virtual reality (VR) environment to study fixation event-related potentials (fERPs) and event-related spectral perturbations (fERSPs) during free exploration behavior. Specifically, the researchers face the following challenges:

1. **Accuracy of Eye Movement Classification**: Recording and classifying eye-tracking data in a 3D virtual reality environment is more complex than in a 2D environment. Existing eye movement classification algorithms are mainly suitable for 2D environments, so there is a need to develop eye movement classification algorithms suitable for 3D VR environments.

2. **Noise Handling**: There is varying degrees of noise in continuously recorded free exploration data, which can affect the accuracy of eye movement classification. Researchers need to find a method to segment the data to reduce the impact of noise.

3. **Correction of Head and Body Movements**: In a VR environment, participants may move their heads and bodies, which can affect the calculation of eye movement speed. Therefore, these movements need to be corrected to ensure the accuracy of eye movement classification.

4. **Synchronization and Alignment of EEG Data**: In free exploration experiments, there are no predefined trials, so eye movement events (such as the start of a fixation) need to be used as event starting points for analyzing fERPs and fERSPs. This requires accurate synchronization and alignment of eye-tracking and EEG data.

By addressing these issues, researchers hope to accurately record and analyze EEG and eye-tracking data in a free exploration virtual reality environment, thereby better understanding cognitive processes in natural scenes.