Abstract:Naturalistic stimuli, including movie, music, and speech, have been increasingly applied in the research of neuroimaging. Relative to a resting-state or single-task state, naturalistic stimuli can evoke more intense brain activities and have been proved to possess higher test-retest reliability, suggesting greater potential to study adaptive human brain function. In the current research, naturalistic functional magnetic resonance imaging (N-fMRI) has been a powerful tool to record brain states under naturalistic stimuli, and many efforts have been devoted to study the high-level semantic features from spatial or temporal representations via N-fMRI. However, integrating both spatial and temporal characteristics of brain activities for better interpreting the patterns under naturalistic stimuli is still underexplored. In this work, a novel hybrid learning framework that comprehensively investigates both the spatial (via Predictive Model) and the temporal [via convolutional neural network (CNN) model] characteristics of the brain is proposed. Specifically, to focus on certain relevant regions from the whole brain, regions of significance (ROS), which contain common spatial activation characteristics across individuals, are selected via the Predictive Model. Further, voxels of significance (VOS), whose signals contain significant temporal characteristics under naturalistic stimuli, are interpreted via one-dimensional CNN (1D-CNN) model. In this article, our proposed framework is applied onto the N-fMRI data during naturalistic classical/pop/speech audios stimuli. The promising performance is achieved via the Predictive Model to differentiate the different audio categories. Especially for distinguishing the classic and speech audios, the accuracy of classification is up to 92%. Moreover, spatial ROS and VOS are effectively obtained. Besides, temporal characteristics of the high-level semantic features are investigated on the frequency domain via convolution kernels of 1D-CNN model, and we effectively bridge the "semantic gap" between high-level semantic features of N-fMRI and low-level acoustic features of naturalistic audios in the frequency domain. Our results provide novel insights on characterizing spatiotemporal patterns of brain activities via N-fMRI and effectively explore the high-level semantic features under naturalistic stimuli, which will further benefit the understanding of the brain working mechanism and the advance of naturalistic stimuli clinical application.

A Unified Multi-Modality Fusion Framework for Deep Spatio-Temporal-Spectral Feature Learning in Resting-State fMRI Denoising

Deep attentive spatio-temporal feature learning for automatic resting-state fMRI denoising

Frequency-Relevant Residual Learning for Multi-Modal Image Denoising.

A Hybrid Spatio-Temporal Deep Belief Network and Sparse Representation-Based Framework Reveals Multi-Level Core Functional Components in Decoding Multi-Task fMRI Signals

MFCA-MICNN: a convolutional neural network with multiscale fast channel attention and multibranch irregular convolution for noise removal in dMRI

Interpretable Fusion Analytics Framework for fMRI Connectivity: Self-Attention Mechanism and Latent Space Item-Response Model

An enhanced network for brain MR image denoising

Integrating spatial and temporal features for enhanced artifact removal in multi-channel EEG recordings

Spatio-Temporal Feature Coupling of Dynamic Functional Connectivity Network for Resting-State fMRI Brain Disease Classification

A hybrid learning framework for fine-grained interpretation of brain spatiotemporal patterns during naturalistic functional magnetic resonance imaging

Multi-Paradigm fMRI Fusion via Sparse Tensor Decomposition in Brain Functional Connectivity Study

Self-supervised learning for denoising of multidimensional MRI data

A robust deep neural network for denoising task-based fMRI data: An application to working memory and episodic memory

A Deep-Learning-Based Multimodal Data Fusion Framework for Urban Region Function Recognition

Multi-Perspective Feature Extraction and Fusion Based on Deep Latent Space for Diagnosis of Alzheimer's Diseases

SpFusionNet: deep learning-driven brain image fusion with spatial frequency analysis

Deepfusion: A Deep Learning Framework For The Fusion Of Heterogeneous Sensory Data

3d Convolutional Long-Short Term Memory Network For Spatiotemporal Modeling Of Fmri Data

A Unified Framework for Static and Dynamic Functional Connectivity Augmentation for Multi-Domain Brain Disorder Classification

Ensemble of expert deep neural networks for spatio-temporal denoising of contrast-enhanced MRI sequences

Completed Feature Disentanglement Learning for Multimodal MRIs Analysis