Abstract:Building computational models to account for the cortical representation of language plays an important role in understanding the human linguistic system. Recent progress in distributed semantic models (DSMs), especially transformer-based methods, has driven advances in many language understanding tasks, making DSM a promising methodology to probe brain language processing. DSMs have been shown to reliably explain cortical responses to word stimuli. However, characterizing the brain activities for sentence processing is much less exhaustively explored with DSMs, especially the deep neural network-based methods. What is the relationship between cortical sentence representations against DSMs? What linguistic features that a DSM catches better explain its correlation with the brain activities aroused by sentence stimuli? Could distributed sentence representations help to reveal the semantic selectivity of different brain areas? We address these questions through the lens of neural encoding and decoding, fueled by the latest developments in natural language representation learning. We begin by evaluating the ability of a wide range of 12 DSMs to predict and decipher the functional magnetic resonance imaging (fMRI) images from humans reading sentences. Most models deliver high accuracy in the left middle temporal gyrus (LMTG) and left occipital complex (LOC). Notably, encoders trained with transformer-based DSMs consistently outperform other unsupervised structured models and all the unstructured baselines. With probing and ablation tasks, we further find that differences in the performance of the DSMs in modeling brain activities can be at least partially explained by the granularity of their semantic representations. We also illustrate the DSM's selectivity for concept categories and show that the topics are represented by spatially overlapping and distributed cortical patterns. Our results corroborate and extend previous findings in understanding t-e relation between DSMs and neural activation patterns and contribute to building solid brain–machine interfaces with deep neural network representations.

Enhancing neural encoding models for naturalistic perception with a multi-level integration of deep neural networks and cortical networks

Upgrading Voxel-wise Encoding Model via Integrated Integration over Features and Brain Networks

Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision

Exploring the Brain-like Properties of Deep Neural Networks: A Neural Encoding Perspective

A Visual Encoding Model Based on Deep Neural Networks and Transfer Learning for Brain Activity Measured by Functional Magnetic Resonance Imaging

Dynamics Based Neural Encoding with Inter-Intra Region Connectivity

A visual encoding model based on deep neural networks and transfer learning

Neural Encoding for Human Visual Cortex With Deep Neural Networks Learning “What” and “Where”

VTVBrain: A Two-stage Brain Encoding Model for Decoding Key Neural Responses in Multimodal Contexts

Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)

Structurally-constrained encoding framework using a multi-voxel reduced-rank latent model for human natural vision

Unidirectional brain-computer interface: Artificial neural network encoding natural images to fMRI response in the visual cortex

Decoding dynamic visual scenes across the brain hierarchy

An Encoding Framework With Brain Inner State for Natural Image Identification

Brain Decodes Deep Nets

Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models

Neural Encoding and Decoding With Distributed Sentence Representations

Visual Encoding and Decoding of the Human Brain Based on Shared Features

Neural encoding and interpretation for high-level visual cortices based on fMRI using image caption features

Encoding brain network response to free viewing of videos

Research on Neural Encoding Models for Biological Vision: Progress and Challenges