Abstract:Building computational models to account for the cortical representation of language plays an important role in understanding the human linguistic system. Recent progress in distributed semantic models (DSMs), especially transformer-based methods, has driven advances in many language understanding tasks, making DSM a promising methodology to probe brain language processing. DSMs have been shown to reliably explain cortical responses to word stimuli. However, characterizing the brain activities for sentence processing is much less exhaustively explored with DSMs, especially the deep neural network-based methods. What is the relationship between cortical sentence representations against DSMs? What linguistic features that a DSM catches better explain its correlation with the brain activities aroused by sentence stimuli? Could distributed sentence representations help to reveal the semantic selectivity of different brain areas? We address these questions through the lens of neural encoding and decoding, fueled by the latest developments in natural language representation learning. We begin by evaluating the ability of a wide range of 12 DSMs to predict and decipher the functional magnetic resonance imaging (fMRI) images from humans reading sentences. Most models deliver high accuracy in the left middle temporal gyrus (LMTG) and left occipital complex (LOC). Notably, encoders trained with transformer-based DSMs consistently outperform other unsupervised structured models and all the unstructured baselines. With probing and ablation tasks, we further find that differences in the performance of the DSMs in modeling brain activities can be at least partially explained by the granularity of their semantic representations. We also illustrate the DSM's selectivity for concept categories and show that the topics are represented by spatially overlapping and distributed cortical patterns. Our results corroborate and extend previous findings in understanding t-e relation between DSMs and neural activation patterns and contribute to building solid brain–machine interfaces with deep neural network representations.

Survey on Distributed Word Embeddings Based on Neural Network Language Models

A Survey On Neural Word Embeddings

Topic Modeling Using Distributed Word Embeddings

Mining Coherent Topics in Documents Using Word Embeddings and Large-Scale Text Data

Word and Document Embeddings based on Neural Network Approaches

Incorporating Linguistic Knowledge for Learning Distributed Word Representations.

From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models

Investigating Language Universal and Specific Properties in Word Embeddings

Pre-Trained Multi-View Word Embedding Using Two-Side Neural Network

Embedding Word Similarity with Neural Machine Translation

A Study on Neural Network Language Modeling

Improving Distributional Similarity with Lessons Learned from Word Embeddings

An Exploration Of Semantic Relations In Neural Word Embeddings Using Extrinsic Knowledge

Distributional Models and Deep Learning Embeddings: Combining the Best of Both Worlds

Measuring Word Significance using Distributed Representations of Words

Text Classification With Document Embeddings

DRWS: A Model for Learning Distributed Representations for Words and Sentences.

A Neurobiologically Motivated Analysis of Distributional Semantic Models

Impact of word embedding models on text analytics in deep learning environment: a review

Recurrent Neural Network Language Model Adaptation Derived Document Vector

Neural Encoding and Decoding With Distributed Sentence Representations