Abstract:Linear sequences of words are implicitly represented in our brains by hierarchical structures that organize the composition of words in sentences. Linguists formalize different frameworks to model this hierarchy; two of the most common syntactic frameworks are Constituency and Dependency. Constituency represents sentences as nested groups of phrases, while dependency represents a sentence by assigning relations between its words. Recently, the pursuit of intelligent machines has produced Language Models (LMs) capable of solving many language tasks with a human-level performance. Many studies now question whether LMs implicitly represent syntactic hierarchies. This thesis focuses on producing constituency and dependency structures from LMs in an unsupervised setting. I review the critical methods in this field and highlight a line of work that utilizes a numerical representation for binary constituency trees (Syntactic Distance). I present a detailed study on StructFormer (SF) (Shen et al., 2021), which retrofits a transformer encoder architecture with a parser network to produce constituency and dependency structures. I present six experiments to analyze and address this field's challenges; experiments include investigating the effect of repositioning the parser network within the SF architecture, evaluating subword-based induced trees, and benchmarking the models developed in the thesis experiments on linguistic tasks. Models benchmarking is performed by participating in the BabyLM challenge, published at CoNLL 2023 (Momen et al., 2023). The results of this thesis encourage further development in the direction of retrofitting transformer-based models to induce syntactic structures, supported by the acceptable performance of SF in different experimental settings and the observed limitations that require innovative solutions to advance the state of syntactic structure induction.

Computational Representation of Linguistic Structures using Domain-Specific Languages

Domain-specific Languages in a Finite Domain Constraint Programming System

MontiCore: a Framework for Compositional Development of Domain Specific Languages

Design and implementation of a domain specific language for phylogenetic inference

MontiCore: A Framework for the Development of Textual Domain Specific Languages

MontiCore: Modular Development of Textual Domain Specific Languages

A Domain Specific Transformation Language

A general semi-structured formalism for computational argumentation: Definition, properties, and examples of application.

A Module System for Domain-Specific Languages

Semantic Construction Grammar: Bridging the NL / Logic Divide

Generating Domain-Specific Transformation Languages for Component & Connector Architecture Descriptions

Modelling constructivist language acquisition through syntactico-semantic pattern finding

From a Natural to a Formal Language with DSL Assistant

Systematically Deriving Domain-Specific Transformation Languages

Design and Implementation of DSL Via Category Theoretic Computations

Metamorphic Domain-Specific Languages: A Journey Into the Shapes of a Language

Really Embedding Domain-Specific Languages into C++

Linguistic Structure Induction from Language Models

Lexicalization and Grammar Development

DisSim: A Discourse-Aware Syntactic Text Simplification Frameworkfor English and German

Exploring Probabilistic Soft Logic as a framework for integrating top-down and bottom-up processing of language in a task context