Abstract:Linking computational natural language processing (NLP) models and neural responses to language in the human brain on the one hand facilitates the effort towards disentangling the neural representations underpinning language perception, on the other hand provides neurolinguistics evidence to evaluate and improve NLP models. Mappings of an NLP model's representations of and the brain activities evoked by linguistic input are typically deployed to reveal this symbiosis. However, two critical problems limit its advancement: 1) The model's representations (artificial neurons, ANs) rely on layer-level embeddings and thus lack fine-granularity; 2) The brain activities (biological neurons, BNs) are limited to neural recordings of isolated cortical unit (i.e., voxel/region) and thus lack integrations and interactions among brain functions. To address those problems, in this study, we 1) define ANs with fine-granularity in transformer-based NLP models (BERT in this study) and measure their temporal activations to input text sequences; 2) define BNs as functional brain networks (FBNs) extracted from functional magnetic resonance imaging (fMRI) data to capture functional interactions in the brain; 3) couple ANs and BNs by maximizing the synchronization of their temporal activations. Our experimental results demonstrate 1) The activations of ANs and BNs are significantly synchronized; 2) the ANs carry meaningful linguistic/semantic information and anchor to their BN signatures; 3) the anchored BNs are interpretable in a neurolinguistic context. Overall, our study introduces a novel, general, and effective framework to link transformer-based NLP models and neural activities in response to language and may provide novel insights for future studies such as brain-inspired evaluation and development of NLP models.

CogBERT: Cognition-Guided Pre-trained Language Models.

Gaze-infused BERT: Do Human Gaze Signals Help Pre-Trained Language Models?

StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding

CogLTX: Applying BERT to Long Texts.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Can humans help BERT gain "confidence"?

CogAlign: Learning to Align Textual Neural Representations to Cognitive Language Processing Signals

BERT4CTR: An Efficient Framework to Combine Pre-trained Language Model with Non-textual Features for CTR Prediction

Semantics-aware BERT for Language Understanding.

ChatGPT-BCI: Word-Level Neural State Classification Using GPT, EEG, and Eye-Tracking Biomarkers in Semantic Inference Reading Comprehension

What Does BERT Look At? An Analysis of BERT's Attention

CoT-BERT: Enhancing Unsupervised Sentence Representation through Chain-of-Thought

C2BERT - Cross-contrast BERT for Chinese Biomedical Sentence Representation.

TOCOL: Improving Contextual Representation of Pre-trained Language Models Via Token-Level Contrastive Learning

CAT-BERT: A Context-Aware Transferable BERT Model for Multi-turn Machine Reading Comprehension.

Intepreting & Improving Pretrained Language Models: A Probabilistic Conceptual Approach

TopicBERT: A Topic-Enhanced Neural Language Model Fine-Tuned for Sentiment Classification

Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models

From Word Embedding to Reading Embedding Using Large Language Model, EEG and Eye-tracking

Coupling Artificial Neurons in BERT and Biological Neurons in the Human Brain