Abstract:Naturalistic stimuli, including movie, music, and speech, have been increasingly applied in the research of neuroimaging. Relative to a resting-state or single-task state, naturalistic stimuli can evoke more intense brain activities and have been proved to possess higher test-retest reliability, suggesting greater potential to study adaptive human brain function. In the current research, naturalistic functional magnetic resonance imaging (N-fMRI) has been a powerful tool to record brain states under naturalistic stimuli, and many efforts have been devoted to study the high-level semantic features from spatial or temporal representations via N-fMRI. However, integrating both spatial and temporal characteristics of brain activities for better interpreting the patterns under naturalistic stimuli is still underexplored. In this work, a novel hybrid learning framework that comprehensively investigates both the spatial (via Predictive Model) and the temporal [via convolutional neural network (CNN) model] characteristics of the brain is proposed. Specifically, to focus on certain relevant regions from the whole brain, regions of significance (ROS), which contain common spatial activation characteristics across individuals, are selected via the Predictive Model. Further, voxels of significance (VOS), whose signals contain significant temporal characteristics under naturalistic stimuli, are interpreted via one-dimensional CNN (1D-CNN) model. In this article, our proposed framework is applied onto the N-fMRI data during naturalistic classical/pop/speech audios stimuli. The promising performance is achieved via the Predictive Model to differentiate the different audio categories. Especially for distinguishing the classic and speech audios, the accuracy of classification is up to 92%. Moreover, spatial ROS and VOS are effectively obtained. Besides, temporal characteristics of the high-level semantic features are investigated on the frequency domain via convolution kernels of 1D-CNN model, and we effectively bridge the "semantic gap" between high-level semantic features of N-fMRI and low-level acoustic features of naturalistic audios in the frequency domain. Our results provide novel insights on characterizing spatiotemporal patterns of brain activities via N-fMRI and effectively explore the high-level semantic features under naturalistic stimuli, which will further benefit the understanding of the brain working mechanism and the advance of naturalistic stimuli clinical application.

Predicting whole-brain neural dynamics from prefrontal cortex fNIRS signal during movie-watching

Mapping fNIRS to fMRI with Neural Data Augmentation and Machine Learning Models

Functional near-infrared spectroscopy imaging of the prefrontal cortex during a naturalistic comedy movie

Identifying Naturalistic Movies from Human Brain Activity with High-Density Diffuse Optical Tomography

Movie-watching outperforms rest for functional connectivity-based prediction of behavior

Prediction of cognitive scores by movie-watching fmri connectivity and eye movement via spectral graph convolutions

Beyond the Surface: Revealing the Depths of Brain Activity by Predicting fMRI from EEG with Deep Learning

A shared neural encoding model for the prediction of subject-specific fMRI response

Inferring Consistent Functional Interaction Patterns from Natural Stimulus FMRI Data

Spatial correspondence of cortical activity measured with whole head fNIRS and fMRI: Toward clinical use within subject

NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties

A New Statistical Approach for fNIRS Hyperscanning to Predict Brain Activity of Preschoolers’ Using Teacher’s

An EEG-informed Hemodynamic Response Modeling Method for Fnirs Signals

Individual differences in functional connectivity during naturalistic viewing conditions

Imaging Brain Function with Functional Near-Infrared Spectroscopy in Unconstrained Environments

Functional Brain Interactions During Free Viewing Of Video Stream

Assessing Brain Networks by Resting-State Dynamic Functional Connectivity: An fNIRS-EEG Study

Taking a Deeper Look at the Brain: Predicting Visual Perceptual and Working Memory Load From High-Density fNIRS Data

A hybrid learning framework for fine-grained interpretation of brain spatiotemporal patterns during naturalistic functional magnetic resonance imaging

Exploring the impact of 3D movie watching on the brain source activities and energy consumption by ESI and fNIRS

Prediction of Cognitive Scores by Joint Use of Movie-Watching fMRI Connectivity and Eye Tracking via Attention-CensNet