Abstract:Functional magnetic resonance imaging (fMRI) is a methodology for measuring human brain activities. It has become more and more popular in neural decoding due to its noninvasive. Neural decoding aims to establishing models to reconstruct external stimuli or features of stimuli from known brain responses, so that we can understand the principles of brain functions such as emotion, cognition and language. Neural decoding based on fMRI is of great significance for further understanding the mechanism of brain operation. Most existing studies take multi-scale topology information of brain networks obtained from fMRI into account in neural decoding. However, they always ignore the simultaneous modeling of network structure and hemodynamic response, thus leading to information loss. In addition, current multi-scale methods usually only utilize spatial or logical reasoning relationship of brain networks, which brings challenge to precise neural decoding. In this work, we present a novel and robust multi-scale spatial and logical reasoning learning framework (MSLR) for fMRI-based neural decoding. Specifically, we first design graph signal wavelet generation module to combine brain network topology and node information to construct multi-scale representation of brain networks in a local to global manner. Then, we develop multi-scale information fusion module that can simultaneously model the spatial and logical reasoning relationship of brain networks, it can also learn discriminative multi-scale features with brain state transition. Finally, we construct a neural decoding module to predict the brain states. We evaluated the framework on the public Human Connectome Project (HCP) dataset that included 986 participants. The experimental results with support vector machine (SVM) outperform current state-of-the-art methods on four evaluation metrics (accuracy: 91.58, kappa coefficient: 0.883, macro F1: 0.865 and hamming distance: 0.105) under 19 different stimuli spanning 7 different cognitive tasks. The interpretation of the learned multi-scale representation replicates neuroscientific findings from previous fMRI studies and renews the multi-scale information flow pattern of brain network in neural decoding.

Multimodal deep neural decoding reveals highly resolved spatiotemporal profile of visual object representation in humans

Long short‐term memory‐based neural decoding of object categories evoked by natural images

‘When’ and ‘what’ did you see? A novel fMRI-based visual decoding framework

Efficient Neural Decoding Based on Multimodal Training

Category Decoding of Visual Stimuli From Human Brain Activity Using a Bidirectional Recurrent Neural Network to Simulate Bidirectional Information Flows in Human Visual Cortices

Deep Neural Networks predict Hierarchical Spatio-temporal Cortical Dynamics of Human Visual Object Recognition

Decoding Visual Feature Versus Visual Spatial Attention Control with Deep Neural Networks

Generic decoding of seen and imagined objects using hierarchical visual features

Convolutional neural network-based encoding and decoding of visual object recognition in space and time

Brain-aligning of semantic vectors improves neural decoding of visual stimuli

Hierarchical Neural Representation of Dreamed Objects Revealed by Brain Decoding with Deep Neural Network Features

Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features

Categorizing Objects from MEG Signals Using EEGNet

Rapidly Decoding Image Categories from MEG Data Using a Multivariate Short-Time FC Pattern Analysis Approach

Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision

Multi-Semantic Decoding of Visual Perception with Graph Neural Networks

Decoding human brain activity with deep learning

Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models

Jointly Fusing Multi-Scale Spatial-Logical Brain Networks: A Neural Decoding Method

Temporal Variabilities Provide Additional Category-Related Information in Object Category Decoding: A Systematic Comparison of Informative EEG Features

Brain decoding: toward real-time reconstruction of visual perception