Abstract:Functional magnetic resonance imaging (fMRI) is a methodology for measuring human brain activities. It has become more and more popular in neural decoding due to its noninvasive. Neural decoding aims to establishing models to reconstruct external stimuli or features of stimuli from known brain responses, so that we can understand the principles of brain functions such as emotion, cognition and language. Neural decoding based on fMRI is of great significance for further understanding the mechanism of brain operation. Most existing studies take multi-scale topology information of brain networks obtained from fMRI into account in neural decoding. However, they always ignore the simultaneous modeling of network structure and hemodynamic response, thus leading to information loss. In addition, current multi-scale methods usually only utilize spatial or logical reasoning relationship of brain networks, which brings challenge to precise neural decoding. In this work, we present a novel and robust multi-scale spatial and logical reasoning learning framework (MSLR) for fMRI-based neural decoding. Specifically, we first design graph signal wavelet generation module to combine brain network topology and node information to construct multi-scale representation of brain networks in a local to global manner. Then, we develop multi-scale information fusion module that can simultaneously model the spatial and logical reasoning relationship of brain networks, it can also learn discriminative multi-scale features with brain state transition. Finally, we construct a neural decoding module to predict the brain states. We evaluated the framework on the public Human Connectome Project (HCP) dataset that included 986 participants. The experimental results with support vector machine (SVM) outperform current state-of-the-art methods on four evaluation metrics (accuracy: 91.58, kappa coefficient: 0.883, macro F1: 0.865 and hamming distance: 0.105) under 19 different stimuli spanning 7 different cognitive tasks. The interpretation of the learned multi-scale representation replicates neuroscientific findings from previous fMRI studies and renews the multi-scale information flow pattern of brain network in neural decoding.

Neural Encoding and Decoding With Distributed Sentence Representations

Delta-band Neural Activity Primarily Tracks Sentences Instead of Semantic Properties of Words

Distributed Representation of Semantics in the Human Brain: Evidence from Studies Using Natural Language Processing Techniques

Towards Linguistic Neural Representation Learning and Sentence Retrieval from Electroencephalogram Recordings

Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)

Decoding Linguistic Representations of Human Brain

Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?

A dual‐channel language decoding from brain activity with progressive transfer training

Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models

Exploring Semantic Representation in Brain Activity Using Word Embeddings.

Enhancing neural encoding models for naturalistic perception with a multi-level integration of deep neural networks and cortical networks

Jointly Fusing Multi-Scale Spatial-Logical Brain Networks: A Neural Decoding Method

Emotion Recognition with Feature Extracted from the Manifold of Brain Networks

Describing Semantic Representations of Brain Activity Evoked by Visual Stimuli

Experiential, Distributional and Dependency-based Word Embeddings have Complementary Roles in Decoding Brain Activity

Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction

Decoding Brain Activity Associated with Literal and Metaphoric Sentence Comprehension Using Distributional Semantic Models

Deep learning models of cognitive processes constrained by human brain connectomes

Brain encoding models based on multimodal transformers can transfer across language and vision

Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)

Decoding speech from non-invasive brain recordings