Abstract:Linear sequences of words are implicitly represented in our brains by hierarchical structures that organize the composition of words in sentences. Linguists formalize different frameworks to model this hierarchy; two of the most common syntactic frameworks are Constituency and Dependency. Constituency represents sentences as nested groups of phrases, while dependency represents a sentence by assigning relations between its words. Recently, the pursuit of intelligent machines has produced Language Models (LMs) capable of solving many language tasks with a human-level performance. Many studies now question whether LMs implicitly represent syntactic hierarchies. This thesis focuses on producing constituency and dependency structures from LMs in an unsupervised setting. I review the critical methods in this field and highlight a line of work that utilizes a numerical representation for binary constituency trees (Syntactic Distance). I present a detailed study on StructFormer (SF) (Shen et al., 2021), which retrofits a transformer encoder architecture with a parser network to produce constituency and dependency structures. I present six experiments to analyze and address this field's challenges; experiments include investigating the effect of repositioning the parser network within the SF architecture, evaluating subword-based induced trees, and benchmarking the models developed in the thesis experiments on linguistic tasks. Models benchmarking is performed by participating in the BabyLM challenge, published at CoNLL 2023 (Momen et al., 2023). The results of this thesis encourage further development in the direction of retrofitting transformer-based models to induce syntactic structures, supported by the acceptable performance of SF in different experimental settings and the observed limitations that require innovative solutions to advance the state of syntactic structure induction.

Assessment of Pre-Trained Models Across Languages and Grammars

Multilingual Syntax-aware Language Modeling through Dependency Tree Conversion

Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Models

How Well Do Large Language Models Understand Syntax? An Evaluation by Asking Natural Language Questions

Cross-Linguistic Syntactic Evaluation of Word Prediction Models

Probing LLMs for Joint Encoding of Linguistic Categories

Linguistic Structure Induction from Language Models

Controlled Evaluation of Syntactic Knowledge in Multilingual Language Models

Seeing Syntax: Uncovering Syntactic Learning Limitations in Vision-Language Models

Large Language Models Demonstrate the Potential of Statistical Learning in Language

Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies

Evaluating Neural Language Models as Cognitive Models of Language Acquisition

Towards Compositionally Generalizable Semantic Parsing in Large Language Models: A Survey

Analyzing Large Language Models for Classroom Discussion Assessment

Large Language Models as Neurolinguistic Subjects: Identifying Internal Representations for Form and Meaning

Exploring syntactic information in sentence embeddings through multilingual subject-verb agreement

Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach

LS-Tree: Model Interpretation When the Data Are Linguistic

Constituency Parsing using LLMs

Large Linguistic Models: Analyzing theoretical linguistic abilities of LLMs

Language models align with human judgments on key grammatical constructions