Abstract:Naturalistic stimuli, including movie, music, and speech, have been increasingly applied in the research of neuroimaging. Relative to a resting-state or single-task state, naturalistic stimuli can evoke more intense brain activities and have been proved to possess higher test-retest reliability, suggesting greater potential to study adaptive human brain function. In the current research, naturalistic functional magnetic resonance imaging (N-fMRI) has been a powerful tool to record brain states under naturalistic stimuli, and many efforts have been devoted to study the high-level semantic features from spatial or temporal representations via N-fMRI. However, integrating both spatial and temporal characteristics of brain activities for better interpreting the patterns under naturalistic stimuli is still underexplored. In this work, a novel hybrid learning framework that comprehensively investigates both the spatial (via Predictive Model) and the temporal [via convolutional neural network (CNN) model] characteristics of the brain is proposed. Specifically, to focus on certain relevant regions from the whole brain, regions of significance (ROS), which contain common spatial activation characteristics across individuals, are selected via the Predictive Model. Further, voxels of significance (VOS), whose signals contain significant temporal characteristics under naturalistic stimuli, are interpreted via one-dimensional CNN (1D-CNN) model. In this article, our proposed framework is applied onto the N-fMRI data during naturalistic classical/pop/speech audios stimuli. The promising performance is achieved via the Predictive Model to differentiate the different audio categories. Especially for distinguishing the classic and speech audios, the accuracy of classification is up to 92%. Moreover, spatial ROS and VOS are effectively obtained. Besides, temporal characteristics of the high-level semantic features are investigated on the frequency domain via convolution kernels of 1D-CNN model, and we effectively bridge the "semantic gap" between high-level semantic features of N-fMRI and low-level acoustic features of naturalistic audios in the frequency domain. Our results provide novel insights on characterizing spatiotemporal patterns of brain activities via N-fMRI and effectively explore the high-level semantic features under naturalistic stimuli, which will further benefit the understanding of the brain working mechanism and the advance of naturalistic stimuli clinical application.

Neural dSCA: demixing multimodal interaction among brain areas during naturalistic experiments

Extracting multidimensional stimulus-response correlations using hybrid encoding-decoding of neural activity

Demixed principal component analysis of neural population data

A New Research Trend in Social Neuroscience: Towards an Interactive-Brain Neuroscience

Intersubject Dynamic Conditional Correlation: A Novel Method to Track the Framewise Network Implication during Naturalistic Stimuli

Exploiting Dimensionality Reduction and Neural Network Techniques for the Development of Expert Brain-Computer Interfaces

Integrated Brain Connectivity Analysis with fMRI, DTI, and sMRI Powered by Interpretable Graph Neural Networks

Neural Processing of Naturalistic Audiovisual Events in Space and Time

Towards Modeling the Interaction of Spatial-Associative Neural Network Representations for Multisensory Perception

Bridging natruralistic stimuli, eye movement and brain activity via cca and locality preserving projection

A Realistic Framework for Investigating Decision Making in the Brain With High Spatiotemporal Resolution Using Simultaneous EEG/fMRI and Joint ICA

Discovering hidden brain network responses to naturalistic stimuli via tensor component analysis of multi-subject fMRI data

A hybrid learning framework for fine-grained interpretation of brain spatiotemporal patterns during naturalistic functional magnetic resonance imaging

NeuralSI: Neural Design of Semantic Interaction for Interactive Deep Learning

Enhancing neural encoding models for naturalistic perception with a multi-level integration of deep neural networks and cortical networks

Multimodal active subspace analysis for computing assessment oriented subspaces from neuroimaging data

Inferring Consistent Functional Interaction Patterns from Natural Stimulus FMRI Data

Multi-faceted Neuroimaging Data Integration via Analysis of Subspaces

DENOISING: Dynamic enhancement and noise overcoming in multimodal neural observations via high-density CMOS-based biosensors

Independent component analysis-based source-level hyperlink analysis for two-person neuroscience studies.