Abstract:Building computational models to account for the cortical representation of language plays an important role in understanding the human linguistic system. Recent progress in distributed semantic models (DSMs), especially transformer-based methods, has driven advances in many language understanding tasks, making DSM a promising methodology to probe brain language processing. DSMs have been shown to reliably explain cortical responses to word stimuli. However, characterizing the brain activities for sentence processing is much less exhaustively explored with DSMs, especially the deep neural network-based methods. What is the relationship between cortical sentence representations against DSMs? What linguistic features that a DSM catches better explain its correlation with the brain activities aroused by sentence stimuli? Could distributed sentence representations help to reveal the semantic selectivity of different brain areas? We address these questions through the lens of neural encoding and decoding, fueled by the latest developments in natural language representation learning. We begin by evaluating the ability of a wide range of 12 DSMs to predict and decipher the functional magnetic resonance imaging (fMRI) images from humans reading sentences. Most models deliver high accuracy in the left middle temporal gyrus (LMTG) and left occipital complex (LOC). Notably, encoders trained with transformer-based DSMs consistently outperform other unsupervised structured models and all the unstructured baselines. With probing and ablation tasks, we further find that differences in the performance of the DSMs in modeling brain activities can be at least partially explained by the granularity of their semantic representations. We also illustrate the DSM's selectivity for concept categories and show that the topics are represented by spatially overlapping and distributed cortical patterns. Our results corroborate and extend previous findings in understanding t-e relation between DSMs and neural activation patterns and contribute to building solid brain–machine interfaces with deep neural network representations.

Describing Semantic Representations of Brain Activity Evoked by Visual Stimuli

Mind captioning: Evolving descriptive text of mental content from human brain activity

MindSemantix: Deciphering Brain Visual Experiences with a Brain-Language Model

Brain Captioning: Decoding human brain activity into images and text

BrainChat: Decoding Semantic Information from fMRI using Vision-language Pretrained Models

Exploring the Relationship Between Visual Information and Language Semantic Concept in the Human Brain

BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity

MindGPT: Interpreting What You See with Non-invasive Brain Recordings

Visual representations in the human brain are aligned with large language models

Decoding Semantics Categorization During Natural Viewing of Video Streams.

Exploring Semantic Representation in Brain Activity Using Word Embeddings.

Emotion Recognition with Feature Extracted from the Manifold of Brain Networks

Unveiling Multi-level and Multi-modal Semantic Representations in the Human Brain using Large Language Models

Neural Encoding and Decoding With Distributed Sentence Representations

Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models

Reading visually embodied meaning from the brain: Visually grounded computational models decode visual-object mental imagery induced by written text

Distributed Representation of Semantics in the Human Brain: Evidence from Studies Using Natural Language Processing Techniques

A neural decoding algorithm that generates language from visual activity evoked by natural images

Decoding human brain activity with deep learning

A dual‐channel language decoding from brain activity with progressive transfer training

UniBrain: Unify Image Reconstruction and Captioning All in One Diffusion Model from Human Brain Activity