Abstract:When we speak, write or listen, we continuously make predictions based on our knowledge of a language's grammar. Remarkably, children acquire this grammatical knowledge within just a few years, enabling them to understand and generalise to novel constructions that have never been uttered before. Language models are powerful tools that create representations of language by incrementally predicting the next word in a sentence, and they have had a tremendous societal impact in recent years. The central research question of this thesis is whether these models possess a deep understanding of grammatical structure similar to that of humans. This question lies at the intersection of natural language processing, linguistics, and interpretability. To address it, we will develop novel interpretability techniques that enhance our understanding of the complex nature of large-scale language models. We approach our research question from three directions. First, we explore the presence of abstract linguistic information through structural priming, a key paradigm in psycholinguistics for uncovering grammatical structure in human language processing. Next, we examine various linguistic phenomena, such as adjective order and negative polarity items, and connect a model's comprehension of these phenomena to the data distribution on which it was trained. Finally, we introduce a controlled testbed for studying hierarchical structure in language models using various synthetic languages of increasing complexity and examine the role of feature interactions in modelling this structure. Our findings offer a detailed account of the grammatical knowledge embedded in language model representations and provide several directions for investigating fundamental linguistic questions using computational methods.

Construction Grammar and Language Models

The Computational Learning of Construction Grammars: State of the Art and Prospective Roadmap

A Constructional Approach to Grammar

Explaining pretrained language models' understanding of linguistic structures using construction grammar

Construction Grammar and Artificial Intelligence

CoELM: Construction-Enhanced Language Modeling

Modelling constructivist language acquisition through syntactico-semantic pattern finding

Enhancing Language Representation with Constructional Information for Natural Language Understanding

Five Ways in Which Computational Modeling Can Help Advance Cognitive Science: Lessons From Artificial Grammar Learning

Syntactic Structure from Deep Learning

Constructions Are Catenae: Construction Grammar Meets Dependency Grammar

Opportunities and Challenges of Generative AI in Construction Industry: Focusing on Adoption of Text-Based Models

Language Models as Models of Language

Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomena

Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons

Opening the black box of language acquisition

Modeling structure-building in the brain with CCG parsing and large language models

The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlative

Toward Cultural Interpretability: A Linguistic Anthropological Framework for Describing and Evaluating Large Language Models (LLMs)

Finding Structure in Language Models

A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spaces