Abstract:Linear sequences of words are implicitly represented in our brains by hierarchical structures that organize the composition of words in sentences. Linguists formalize different frameworks to model this hierarchy; two of the most common syntactic frameworks are Constituency and Dependency. Constituency represents sentences as nested groups of phrases, while dependency represents a sentence by assigning relations between its words. Recently, the pursuit of intelligent machines has produced Language Models (LMs) capable of solving many language tasks with a human-level performance. Many studies now question whether LMs implicitly represent syntactic hierarchies. This thesis focuses on producing constituency and dependency structures from LMs in an unsupervised setting. I review the critical methods in this field and highlight a line of work that utilizes a numerical representation for binary constituency trees (Syntactic Distance). I present a detailed study on StructFormer (SF) (Shen et al., 2021), which retrofits a transformer encoder architecture with a parser network to produce constituency and dependency structures. I present six experiments to analyze and address this field's challenges; experiments include investigating the effect of repositioning the parser network within the SF architecture, evaluating subword-based induced trees, and benchmarking the models developed in the thesis experiments on linguistic tasks. Models benchmarking is performed by participating in the BabyLM challenge, published at CoNLL 2023 (Momen et al., 2023). The results of this thesis encourage further development in the direction of retrofitting transformer-based models to induce syntactic structures, supported by the acceptable performance of SF in different experimental settings and the observed limitations that require innovative solutions to advance the state of syntactic structure induction.

Optimality Theory with Candidate Chains:A Survey

Optimality Theory with Candidate Chains:Problem and Solution

Optimality Theory and Phonological Opacity

Optimality Theory:Extended application and developmental trend

Beyond the Constituent - A Dependency Grammar Analysis of Chains

Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMs

Optimality Theory as a Framework for Lexical Acquisition

Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs

Generating Chain-of-Thoughts with a Pairwise-Comparison Approach to Searching for the Most Promising Intermediate Thought

Align With Purpose: Optimize Desired Properties in CTC Models with a General Plug-and-Play Framework

Chain-of-Thought in Large Language Models: Decoding, Projection, and Activation

Switching Auxiliary Chains for Speech Recognition

Supervised Chain of Thought

Constrained Reasoning Chains for Enhancing Theory-of-Mind in Large Language Models

Chain-of-Probe: Examing the Necessity and Accuracy of CoT Step-by-Step

Markovian Transformers for Informative Language Modeling

Linguistic Structure Induction from Language Models

A Theoretical Understanding of Chain-of-Thought: Coherent Reasoning and Error-Aware Demonstration

Acoustic-To-Word Model Without OOV

Chain-of-Thoughts for Molecular Understanding

Serialism and Opacity in Phonological Theory