Abstract:Multivoxel pattern analysis (MVPA) examines the differences in multivoxel patterns evoked by different cognitive conditions using machine learning methods such as logistic regression and support vector machine. These methods are linear methods. It is possible that there are nonlinear relationships in the data that are not readily detected by these methods. We attempted to apply deep neural networks to address this problem. FMRI data were recorded from humans (n=20) performing a cued visual spatial/feature attention task in which an auditory cue instructed the subject to attend either left or right visual field (spatial trials), or attend either red or green color (feature trials). Following a random delay, two rectangular stimuli appeared, one in each visual field, and the subjects reported the orientation of the rectangle in the attended location (spatial trials) or with the attended color (feature trials). A deep neural network (DNN) was trained to take cue-evoked fMRI data as input features to predict trial labels. For feature (spatial) attention control, feature (spatial) trial data from 19 subjects were used to train a DNN model, which was then tested on the remaining subject. This process was repeated 20 times and the 20 decoding accuracies were averaged. We found that using the whole brain, the accuracy for decoding feature attention (cue red vs cue green) and spatial attention control (cue left vs cue right) is 59% and 61% respectively, both significantly above chance level of 50%. The heatmaps derived from the DNN models revealed important regions that contribute to both feature and spatial attention control as well as regions that contribute mainly to feature attention control or spatial attention control. In sum, DNNs can yield insights underlying attention control that complement other methods and provide a new approach for uncovering more complex relations between cognitive conditions and neural activities.

AlignedCut: Visual Concepts Discovery on Brain-Guided Universal Feature Space

Finding Shared Decodable Concepts and their Negations in the Brain

Brain Decodes Deep Nets

Exploring Brain Effective Connectivity in Visual Perception Using a Hierarchical Correlation Network.

Identifying Shared Decodable Concepts in the Human Brain Using Image-Language Foundation Models

Bridging the Semantic Latent Space Between Brain and Machine: Similarity is All You Need

Parallel Backpropagation for Shared-Feature Visualization

One Hundred Neural Networks and Brains Watching Videos: Lessons from Alignment

Visual Encoding and Decoding of the Human Brain Based on Shared Features

Revealing Vision-Language Integration in the Brain with Multimodal Networks

Deciphering 'What' and 'Where' Visual Pathways from Spectral Clustering of Layer-Distributed Neural Representations

Vision CNNs trained to estimate spatial latents learned similar ventral-stream-aligned representations

Integrated deep visual and semantic attractor neural networks predict fMRI pattern-information along the ventral object processing pathway

Hierarchical Bayesian Causality Network to Extract High-Level Semantic Information in Visual Cortex

Teaching CORnet Human fMRI Representations for Enhanced Model-Brain Alignment

Aligning brain functions boosts the decoding of visual semantics in novel subjects

Visual Concept Connectome (VCC): Open World Concept Discovery and their Interlayer Connections in Deep Models

Unidirectional brain-computer interface: Artificial neural network encoding natural images to fMRI response in the visual cortex

Decoding Visual Feature Versus Visual Spatial Attention Control with Deep Neural Networks

Complex Properties of Training Stimuli Affect Brain Alignment in a Deep Network Model of Mouse Visual Cortex