Abstract:Naturalistic stimuli, including movie, music, and speech, have been increasingly applied in the research of neuroimaging. Relative to a resting-state or single-task state, naturalistic stimuli can evoke more intense brain activities and have been proved to possess higher test-retest reliability, suggesting greater potential to study adaptive human brain function. In the current research, naturalistic functional magnetic resonance imaging (N-fMRI) has been a powerful tool to record brain states under naturalistic stimuli, and many efforts have been devoted to study the high-level semantic features from spatial or temporal representations via N-fMRI. However, integrating both spatial and temporal characteristics of brain activities for better interpreting the patterns under naturalistic stimuli is still underexplored. In this work, a novel hybrid learning framework that comprehensively investigates both the spatial (via Predictive Model) and the temporal [via convolutional neural network (CNN) model] characteristics of the brain is proposed. Specifically, to focus on certain relevant regions from the whole brain, regions of significance (ROS), which contain common spatial activation characteristics across individuals, are selected via the Predictive Model. Further, voxels of significance (VOS), whose signals contain significant temporal characteristics under naturalistic stimuli, are interpreted via one-dimensional CNN (1D-CNN) model. In this article, our proposed framework is applied onto the N-fMRI data during naturalistic classical/pop/speech audios stimuli. The promising performance is achieved via the Predictive Model to differentiate the different audio categories. Especially for distinguishing the classic and speech audios, the accuracy of classification is up to 92%. Moreover, spatial ROS and VOS are effectively obtained. Besides, temporal characteristics of the high-level semantic features are investigated on the frequency domain via convolution kernels of 1D-CNN model, and we effectively bridge the "semantic gap" between high-level semantic features of N-fMRI and low-level acoustic features of naturalistic audios in the frequency domain. Our results provide novel insights on characterizing spatiotemporal patterns of brain activities via N-fMRI and effectively explore the high-level semantic features under naturalistic stimuli, which will further benefit the understanding of the brain working mechanism and the advance of naturalistic stimuli clinical application.

Detection of Brain Network Communities During Natural Speech Comprehension From Functionally Aligned EEG Sources

Temporal-Spatial-Spectral Investigation of Brain Network Dynamics in Human Speech Perception

Study on neural entrainment to continuous speech using dynamic source connectivity analysis

Dynamic Community Detection for Brain Functional Networks During Music Listening with Block Component Analysis.

Distinct Patterns of Functional Connectivity During the Comprehension of Natural, Narrative Speech.

Tripartite organization of brain state dynamics underlying spoken narrative comprehension

Brain Network Reorganization During Visual Search Task Revealed by a Network Analysis of Fixation-Related Potential.

Time-domain Analysis of Neural Tracking of Hierarchical Linguistic Structures

EEG Evidence for a Three-Phase Recurrent Process During Spoken Word Processing.

Unraveling Predictive Mechanism in Speech Perception and Production: Insights from EEG Analyses of Brain Network Dynamics

Cluster Structure and Localization of Brain Functional Networks Based on the ERP Signals of Auditory Task

Community Detection in Multi-frequency EEG Networks

Uncovering Cortical Activations of Discourse Comprehension and Their Overlaps with Common Large-Scale Neural Networks.

A hybrid learning framework for fine-grained interpretation of brain spatiotemporal patterns during naturalistic functional magnetic resonance imaging

Task-modulated Activation and Functional Connectivity of the Temporal and Frontal Areas During Speech Comprehension

Speech Activity Detection from Stereotactic EEG

Exploring Auditory Network Composition During Free Listening to Audio Excerpts Via Group-Wise Sparse Representation.

Functional connectivity of stimulus-evoked brain responses to natural speech in post-stroke aphasia

Data-driven analysis of functional brain interactions during free listening to music and speech

Investigation of Relation Between Speech Perception and Production Based on EEG Source Reconstruction

Effect of Inverse Solutions, Connectivity Measures, and Node Sizes on EEG Source Network: A Simultaneous EEG Study