Abstract:Functional magnetic resonance imaging (fMRI) is a methodology for measuring human brain activities. It has become more and more popular in neural decoding due to its noninvasive. Neural decoding aims to establishing models to reconstruct external stimuli or features of stimuli from known brain responses, so that we can understand the principles of brain functions such as emotion, cognition and language. Neural decoding based on fMRI is of great significance for further understanding the mechanism of brain operation. Most existing studies take multi-scale topology information of brain networks obtained from fMRI into account in neural decoding. However, they always ignore the simultaneous modeling of network structure and hemodynamic response, thus leading to information loss. In addition, current multi-scale methods usually only utilize spatial or logical reasoning relationship of brain networks, which brings challenge to precise neural decoding. In this work, we present a novel and robust multi-scale spatial and logical reasoning learning framework (MSLR) for fMRI-based neural decoding. Specifically, we first design graph signal wavelet generation module to combine brain network topology and node information to construct multi-scale representation of brain networks in a local to global manner. Then, we develop multi-scale information fusion module that can simultaneously model the spatial and logical reasoning relationship of brain networks, it can also learn discriminative multi-scale features with brain state transition. Finally, we construct a neural decoding module to predict the brain states. We evaluated the framework on the public Human Connectome Project (HCP) dataset that included 986 participants. The experimental results with support vector machine (SVM) outperform current state-of-the-art methods on four evaluation metrics (accuracy: 91.58, kappa coefficient: 0.883, macro F1: 0.865 and hamming distance: 0.105) under 19 different stimuli spanning 7 different cognitive tasks. The interpretation of the learned multi-scale representation replicates neuroscientific findings from previous fMRI studies and renews the multi-scale information flow pattern of brain network in neural decoding.

JGAT: a joint spatio-temporal graph attention model for brain decoding

Brainnetformer: Decoding Brain Cognitive States with Spatial-Temporal Cross Attention

Modeling spatio-temporal patterns of holistic functional brain networks via multi-head guided attention graph neural networks (Multi-Head GAGNNs)

A Guided Attention 4D Convolutional Neural Network for Modeling Spatio-Temporal Patterns of Functional Brain Networks.

Temporal Spiking Generative Adversarial Networks for Heading Direction Decoding

DGST-Former: Dynamic Graph with Spatio-Temporal Transpose-Transformer for Brain fMRI Data

GCNs-Net: A Graph Convolutional Neural Network Approach for Decoding Time-resolved EEG Motor Imagery Signals

Attention module improves both performance and interpretability of four‐dimensional functional magnetic resonance imaging decoding neural network

Jointly Fusing Multi-Scale Spatial-Logical Brain Networks: A Neural Decoding Method

Attention module improves both performance and interpretability of 4D fMRI decoding neural network

Multi-Scale Spatio-Temporal Fusion with Adaptive Brain Topology Learning for Fmri Based Neural Decoding.

Attend and Decode: 4D fMRI Task State Decoding Using Attention Models

Attention-based fusion of multiple graphheat networks for structural to functional brain mapping

Functional annotation of human cognitive states using deep graph convolution

A Spatio-Temporal Interactive Attention Network for Motor Imagery EEG Decoding

Prediction of Cognitive Scores by Joint Use of Movie-Watching fMRI Connectivity and Eye Tracking via Attention-CensNet

Visual Image Decoding of Brain Activities Using a Dual Attention Hierarchical Latent Generative Network with Multiscale Feature Fusion

Joint subject-identification and task-decoding from inferred functional brain graphs via a multi-task neural network

FMRI-Guided Time-Symmetric Joint Model for Visual Attention Prediction.

BrainGNN: Interpretable Brain Graph Neural Network for fMRI Analysis

Spatio-Temporal Graph Convolution for Resting-State fMRI Analysis