Abstract:Linear sequences of words are implicitly represented in our brains by hierarchical structures that organize the composition of words in sentences. Linguists formalize different frameworks to model this hierarchy; two of the most common syntactic frameworks are Constituency and Dependency. Constituency represents sentences as nested groups of phrases, while dependency represents a sentence by assigning relations between its words. Recently, the pursuit of intelligent machines has produced Language Models (LMs) capable of solving many language tasks with a human-level performance. Many studies now question whether LMs implicitly represent syntactic hierarchies. This thesis focuses on producing constituency and dependency structures from LMs in an unsupervised setting. I review the critical methods in this field and highlight a line of work that utilizes a numerical representation for binary constituency trees (Syntactic Distance). I present a detailed study on StructFormer (SF) (Shen et al., 2021), which retrofits a transformer encoder architecture with a parser network to produce constituency and dependency structures. I present six experiments to analyze and address this field's challenges; experiments include investigating the effect of repositioning the parser network within the SF architecture, evaluating subword-based induced trees, and benchmarking the models developed in the thesis experiments on linguistic tasks. Models benchmarking is performed by participating in the BabyLM challenge, published at CoNLL 2023 (Momen et al., 2023). The results of this thesis encourage further development in the direction of retrofitting transformer-based models to induce syntactic structures, supported by the acceptable performance of SF in different experimental settings and the observed limitations that require innovative solutions to advance the state of syntactic structure induction.

Strained tetragonal states and Bain paths in metals

Rule-Based and Word-Level Statistics-Based Processing of Language: Insights from Neuroscience

Learning to Predict Novel Noun-Noun Compounds

Language learning and innateness: some implications of Compounds Research

Building Probabilistic Models for Natural Language

Syntactic Complexity Development in the Writings of EFL Learners: Insights from a Dependency Syntactically-Annotated Corpus

Capturing the Heterogeneity of Word Learners by Analyzing Persons

Evaluating statistical language models as pragmatic reasoners

Large Language Models Demonstrate the Potential of Statistical Learning in Language

Word learning and the acquisition of syntactic--semantic overhypotheses

Language Acquisition in Computers

Modelling constructivist language acquisition through syntactico-semantic pattern finding

Linguistic Structure Induction from Language Models

Explaining Large Language Model-Based Neural Semantic Parsers (Student Abstract)

Large Language Models as Neurolinguistic Subjects: Identifying Internal Representations for Form and Meaning

Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training

The probabilistic analysis of language acquisition: Theoretical, computational, and experimental analysis

Augmented linguistic analysis skills: Machine translation and generative AI as pedagogical aids for analyzing complex English compounds

Grammar-based Game Description Generation using Large Language Models

Learning Language Structures through Grounding

Evaluating Neural Language Models as Cognitive Models of Language Acquisition