Abstract:The naturalistic viewing of a video clip enables participants to obtain more information from the clip compared to conventional viewing of a static image. Because changing the field-of-view (FoV) allows new visual information to be obtained, we were motivated to investigate whether naturalistic viewing with varying FoV based on active eye movement can enhance the viewing experience of natural stimuli, such as those found in a video clip with a 360° FoV in an MRI scanner. To this end, we developed a novel naturalistic viewing paradigm based on real-time eye-gaze tracking while participants were watching a 360° panoramic video during fMRI acquisition. The gaze position of the participants was recorded using an eye-tracking computer and then transmitted to a stimulus presentation computer via a TCP/IP connection. The identified gaze position was then used to alter the participants' FoV of the video clip in real-time, so the participants could change their FoV to fully explore the 360° video clip (referred to in this paper as active viewing). The gaze position of one participant while watching a video was used to change the FoV of the same video clip for a paired participant (referred to as yoked or passive viewing). Four 360° panoramic videos were used as stimuli, divided into categories based on the brightness level (i.e., bright vs. dark) and location (i.e., nature vs. city). Each of the subjects participated in the active viewing of one of the two nature videos and one of the two city videos and then engaged in the passive viewing of the other video in each category, followed by conventional viewing with a fixed FoV (referred to as fixed viewing) after each of the active or passive viewings. Forty-eight healthy volunteers participated in the study, and data from 42 of these participants were used in the analysis. Representational similarity analysis (RSA) was conducted in a multiple regression framework using representational dissimilarity matrix (RDM) codes to accommodate all of the information regarding neuronal activations from fMRI analysis and the participants' subjective ratings of their viewing experience with the four video clips and with the two contrasting viewing conditions (i.e., "active-fixed" and "passive-fixed"). It was found that the participants' naturalistic viewing experience of the video clips was substantially more immersive with active viewing than with passive and fixed viewing. The RSA using the RDM codes revealed the brain regions associated with the viewing experience, including eye movement and spatial navigation in the superior frontal area (of Brodmann's area 6) and the inferior/superior parietal areas, respectively. Brain regions potentially associated with cognitive and affective processing during the viewing of the video, such as the default-mode networks and insular/Rolandic operculum areas, were also identified. To the best of our knowledge, this is the first study that has used the participants' eye movements to interactively change their FoV for 360° panoramic video clips in real-time. Our method of utilizing the MRI environment can be further extended to other environments such as electroencephalography and behavioral research. It would also be feasible to apply our method to virtual reality and/or augmented reality systems to maximize user experience based on their eye movement.

A high-density diffuse optical tomography dataset of naturalistic viewing

Identifying Naturalistic Movies from Human Brain Activity with High-Density Diffuse Optical Tomography

Mapping brain function in adults and young children during naturalistic viewing with high-density diffuse optical tomography

A naturalistic neuroimaging database for understanding the brain using ecological stimuli

An open-access dataset of naturalistic viewing using simultaneous EEG-fMRI

High-density diffuse optical tomography for imaging human brain function

A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence

Individual differences in functional connectivity during naturalistic viewing conditions

A multimodal fMRI dataset unifying naturalistic processes with a rich array of experimental tasks

Time-Domain Diffuse Optical Tomography for Precision Neuroscience

A naturalistic viewing paradigm using 360° panoramic video clips and real-time field-of-view changes with eye-gaze tracking

Caltech Conte Center, a multimodal data resource for exploring social cognition and decision-making

Mapping neural correlates of biological motion perception in autistic children using high-density diffuse optical tomography

Modeling short visual events through the BOLD moments video fMRI dataset and metadata

Individual Brain Charting dataset extension, third release for movie watching and retinotopy data

Global motion detection and censoring in high-density diffuse optical tomography

Diffuse optical imaging of the whole head

The Human Connectome Project 7 Tesla retinotopy dataset: Description and population receptive field analysis

Functional near-infrared spectroscopy imaging of the prefrontal cortex during a naturalistic comedy movie

A studyforrest extension, retinotopic mapping and localization of higher visual areas

Release of cognitive and multimodal MRI data including real-world tasks and hippocampal subfield segmentations