Abstract:Several popular Transformer based language models have been found to be successful for text-driven brain encoding. However, existing literature leverages only pretrained text Transformer models and has not explored the efficacy of task-specific learned Transformer representations. In this work, we explore transfer learning from representations learned for ten popular natural language processing tasks (two syntactic and eight semantic) for predicting brain responses from two diverse datasets: Pereira (subjects reading sentences from paragraphs) and Narratives (subjects listening to the spoken stories). Encoding models based on task features are used to predict activity in different regions across the whole brain. Features from coreference resolution, NER, and shallow syntax parsing explain greater variance for the reading activity. On the other hand, for the listening activity, tasks such as paraphrase generation, summarization, and natural language inference show better encoding performance. Experiments across all 10 task representations provide the following cognitive insights: (i) language left hemisphere has higher predictive brain activity versus language right hemisphere, (ii) posterior medial cortex, temporo-parieto-occipital junction, dorsal frontal lobe have higher correlation versus early auditory and auditory association cortex, (iii) syntactic and semantic tasks display a good predictive performance across brain regions for reading and listening stimuli resp.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: Which natural language processing tasks are most predictive of fMRI brain activity? Specifically, the researchers explored the representations learned from ten popular natural language processing tasks (two syntactic tasks and eight semantic tasks) for predicting brain responses from two different datasets: the Pereira dataset (subjects read sentences) and the Narratives dataset (subjects listen to stories). By using an encoding model based on task features to predict the activity of different regions of the brain, the researchers aimed to reveal which NLP tasks are most effective in predicting brain responses in reading and auditory activities. ### Main Contributions 1. **Task Selection and Evaluation**: Given Transformer models fine - tuned for various NLP tasks, the researchers proposed the question of which tasks are most predictive of fMRI brain activity, especially in reading and story - listening tasks. 2. **Task Performance**: - **Reading Task**: Coreference Resolution (CR), Named Entity Recognition (NER), and Shallow Syntax Parsing (SS) have higher predictive performance when reading text. - **Listening - to - Stories Task**: Paraphrase Detection (PD), Summarization (Sum), and Natural Language Inference (NLI) show better correlations when listening to stories. ### Experimental Methods - **Datasets**: The study used the Pereira dataset (subjects read sentences) and the Narratives - Pieman dataset (subjects listen to stories). - **Model**: A Ridge regression model was used to predict brain responses, and 2V2 accuracy and Pearson correlation coefficient were calculated as evaluation metrics. - **Feature Space**: Implicit space features were extracted from ten NLP tasks, including coreference resolution, named entity recognition, natural language inference, paraphrase detection, question answering, sentiment analysis, semantic role labeling, shallow syntax parsing, summarization, and word - sense disambiguation. ### Results - **Reading Task**: CR, NER, SRL, and SS show higher correlations when predicting brain responses. In particular, the left - hemisphere language processing region (Language_LH) has higher encoding performance than the right - hemisphere (Language_RH). - **Listening - to - Stories Task**: PD, Sum, and NLI show higher correlations when predicting brain responses. In particular, the bilateral posterior medial cortex (PMC), which is related to high - level language functions, shows the highest correlation. ### Cognitive Insights - **Left - Right Hemisphere Differences**: The left hemisphere has higher predictive performance in language processing, which is consistent with the known left - hemisphere language dominance. - **Brain Region Differences**: The posterior medial cortex, temporoparietal - occipital junction, and dorsal frontal lobe show higher correlations when predicting brain responses. ### Conclusions There are significant differences in the predictive ability of different language tasks for brain activity in reading and listening - to - stories tasks. Shallow tasks such as NER and SS are more effective in reading tasks, while complex NLP tasks such as PD, Sum, and NLI are more effective in listening - to - stories tasks. These findings are helpful for understanding the mechanisms of the brain in processing different types of language stimuli.

Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?

Neural Encoding and Decoding With Distributed Sentence Representations

fMRI predictors based on language models of increasing complexity recover brain left lateralization

Language Reconstruction with Brain Predictive Coding from fMRI Data

Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)

Explaining neural activity in human listeners with deep learning via natural language processing of narrative text

Emotion Recognition with Feature Extracted from the Manifold of Brain Networks

What Are Large Language Models Mapping to in the Brain? A Case Against Over-Reliance on Brain Scores

Navigating Brain Language Representations: A Comparative Analysis of Neural Language Models and Psychologically Plausible Models

Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)

Syntactic Structure Processing in the Brain while Listening

Visio-Linguistic Brain Encoding

Brains and algorithms partially converge in natural language processing

Experiential, Distributional and Dependency-based Word Embeddings have Complementary Roles in Decoding Brain Activity

Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction

Study on the Connectivity of Language Network in Word Reading and Object Recognition Based on tfMRI

ALIGNING BRAINS INTO A SHARED SPACE IMPROVES THEIR ALIGNMENT TO LARGE LANGUAGE MODELS

Decoding task specific and task general functional architectures of the brain

Transfer learning to decode brain states reflecting the relationship between cognitive tasks

On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior

Driving and suppressing the human language network using large language models