Abstract:Functional magnetic resonance imaging (fMRI) is a methodology for measuring human brain activities. It has become more and more popular in neural decoding due to its noninvasive. Neural decoding aims to establishing models to reconstruct external stimuli or features of stimuli from known brain responses, so that we can understand the principles of brain functions such as emotion, cognition and language. Neural decoding based on fMRI is of great significance for further understanding the mechanism of brain operation. Most existing studies take multi-scale topology information of brain networks obtained from fMRI into account in neural decoding. However, they always ignore the simultaneous modeling of network structure and hemodynamic response, thus leading to information loss. In addition, current multi-scale methods usually only utilize spatial or logical reasoning relationship of brain networks, which brings challenge to precise neural decoding. In this work, we present a novel and robust multi-scale spatial and logical reasoning learning framework (MSLR) for fMRI-based neural decoding. Specifically, we first design graph signal wavelet generation module to combine brain network topology and node information to construct multi-scale representation of brain networks in a local to global manner. Then, we develop multi-scale information fusion module that can simultaneously model the spatial and logical reasoning relationship of brain networks, it can also learn discriminative multi-scale features with brain state transition. Finally, we construct a neural decoding module to predict the brain states. We evaluated the framework on the public Human Connectome Project (HCP) dataset that included 986 participants. The experimental results with support vector machine (SVM) outperform current state-of-the-art methods on four evaluation metrics (accuracy: 91.58, kappa coefficient: 0.883, macro F1: 0.865 and hamming distance: 0.105) under 19 different stimuli spanning 7 different cognitive tasks. The interpretation of the learned multi-scale representation replicates neuroscientific findings from previous fMRI studies and renews the multi-scale information flow pattern of brain network in neural decoding.

Analysis of Functional Areas of Human Brain Based on Reconstructed Images of DMFG-generated Countermeasure Network

Deep Natural Image Reconstruction from Human Brain Activity Based on Conditional Progressively Growing Generative Adversarial Networks

Deep image reconstruction from human brain activity

Functional diversity of visual cortex improves constraint-free natural image reconstruction from human brain activity

Reconstruction of Natural Images from Human fMRI Using a Three-Stage Multi-Level Deep Fusion Model

Natural Image Reconstruction from fMRI Based on Node–Edge Interaction and Multi–Scale Constraint

Brain3D: Generating 3D Objects from fMRI

Decoding human brain activity with deep learning

Pre-training Deep Neural Network for Decoding Multiple Brain Regions Can Enhance Image Reconstruction

Jointly Fusing Multi-Scale Spatial-Logical Brain Networks: A Neural Decoding Method

Cortical surface based identification of brain networks using high spatial resolution resting state FMRI data

Brainformer: Mimic Human Visual Brain Functions to Machine Vision Models via fMRI

Constructing Fine-Granularity Functional Brain Network Atlases Via Deep Convolutional Autoencoder.

Brain Decoding Using Deep Convolutional Network and Its Application in Cross-Subject Analysis.

MindLDM: Reconstruct Visual Stimuli from Fmri Using Latent Diffusion Model

Visual Image Decoding of Brain Activities Using a Dual Attention Hierarchical Latent Generative Network with Multiscale Feature Fusion

Big Data Analysis of the Human Brain'S Functional Interactions Based on Fmri

[Brain Functional Network Reconstruction Based on Compressed Sensing and Fast Iterative Shrinkage-Thresholding Algorithm].

Visual Image Decoding of Brain Activities using a Dual Attention Hierarchical Latent Generative Network with Multi-Scale Feature Fusion

DeepBrainSeg: Automated Brain Region Segmentation for Micro-Optical Images with a Convolutional Neural Network

Study on the Application of Mrf and the D-S Theory to Image Segmentation of the Human Brain and Quantitative Analysis of the Brain Tissue