Abstract:We explore which linguistic factors -- at the sentence and token level -- play an important role in influencing language model predictions, and investigate whether these are reflective of results found in humans and human corpora (Gries and Kootstra, 2017). We make use of the structural priming paradigm, where recent exposure to a structure facilitates processing of the same structure. We don't only investigate whether, but also where priming effects occur, and what factors predict them. We show that these effects can be explained via the inverse frequency effect, known in human priming, where rarer elements within a prime increase priming effects, as well as lexical dependence between prime and target. Our results provide an important piece in the puzzle of understanding how properties within their context affect structural prediction in language models.

What problem does this paper attempt to address?

The paper attempts to address the question of whether language models exhibit human-like structural priming effects. Specifically, the authors investigate which linguistic factors significantly influence the predictions of language models at the sentence and lexical levels, and whether these factors align with results observed in humans and human corpora. The study employs the structural priming paradigm to explore the priming effects of language models when processing specific structures, as well as the locations and predictive factors of these effects. ### Main Research Questions: 1. **Do language models exhibit structural priming effects?** - The authors experimentally verify whether language models are influenced by prior exposure to the same structure when processing specific structures. 2. **What are the locations and predictive factors of the priming effects?** - The authors not only investigate the existence of priming effects but also explore their specific locations within sentences and the factors that can predict these effects. 3. **Are the priming effects in language models similar to those in humans?** - The authors compare the structural priming effects in language models and humans, focusing particularly on factors such as the inverse frequency effect and lexical dependence. ### Research Background: - **Structural Priming** refers to the phenomenon where speakers are more likely to reuse a structure they have recently encountered. This phenomenon has been extensively studied in human language production and comprehension. - **Inverse Frequency Effect** refers to the observation in human priming effects that rare elements enhance priming effects, and shared vocabulary also enhances priming effects. - **Lexical Dependence** indicates that priming effects are influenced by lexical overlap and semantic similarity. ### Experimental Design: - **Dataset**: The experiments use dative constructions from the Prime-LM corpus. - **Models**: Several large language models are considered, such as GPT2-large, Llama-2, Falcon-7b, etc. - **Metrics**: Sentence-level priming effects (s-PE) and lexical-level priming effects (w-PE) are used to evaluate the priming effects. ### Main Findings: - **Asymmetry of Priming Effects**: Priming effects in language models typically exhibit asymmetry, meaning some structures are more easily primed than others. - **Impact of Lexical Overlap**: Increasing lexical overlap, especially of verbs and function words, can balance the priming effects, making them more symmetrical. - **Inverse Frequency Effect**: The priming effects in language models are similar to the inverse frequency effect in humans, where rare structures enhance priming effects. ### Conclusion: This study provides important insights into how language models handle structural predictions in context, particularly revealing the impact of lexical overlap and the inverse frequency effect on priming effects. These findings contribute to a deeper understanding of the mechanisms of language models in language comprehension and generation.

Do Language Models Exhibit Human-like Structural Priming Effects?

Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations

Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Models

Finding Structure in Language Models

Do structure predictions persevere to multilinguals' other languages? Evidence from cross-linguistic structural priming in comprehension

Modeling Bilingual Sentence Processing: Evaluating RNN and Transformer Architectures for Cross-Language Structural Priming

Does injecting linguistic structure into language models lead to better alignment with brain recordings?

Large Language Models Are Partially Primed in Pronoun Interpretation

A Review of Structural Priming

How do phonology and orthography feed back to influence syntactic encoding in language production? Evidence from structural priming in Mandarin

Structural Priming in Language Comprehension Studies

Do structural priming and inverse preference effect demand cognitive resources? Evidence from structural priming in production

The Primacy of Experience in Language Processing: Semantic Priming Is Driven Primarily by Experiential Similarity

Are there segmental and tonal effects on syntactic encoding? Evidence from structural priming in Mandarin

Individual differences in structural priming in bilingual and monolingual children: the influence of perspective-taking

Structural Priming and Frequency Effects Interact in Chinese Sentence Comprehension

SPAWNing Structural Priming Predictions from a Cognitively Motivated Parser

Error-based structure prediction in language comprehension: Evidence from verb bias effects in a visual-world structural priming paradigm for Mandarin Chinese.

A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humans

Abstractness, Allomorphy, and Lexical Architecture

The role of explicit memory in syntactic persistence: Effects of lexical cueing and load on sentence memory and sentence production