Abstract:Recurrent Neural Networks (RNNs) trained on a language modeling task have been shown to acquire a number of non-local grammatical dependencies with some success. Here, we provide new evidence that RNN language models are sensitive to hierarchical syntactic structure by investigating the filler--gap dependency and constraints on it, known as syntactic islands. Previous work is inconclusive about whether RNNs learn to attenuate their expectations for gaps in island constructions in particular or in any sufficiently complex syntactic environment. This paper gives new evidence for the former by providing control studies that have been lacking so far. We demonstrate that two state-of-the-art RNN models are are able to maintain the filler--gap dependency through unbounded sentential embeddings and are also sensitive to the hierarchical relationship between the filler and the gap. Next, we demonstrate that the models are able to maintain possessive pronoun gender expectations through island constructions---this control case rules out the possibility that island constructions block all information flow in these networks. We also evaluate three untested islands constraints: coordination islands, left branch islands, and sentential subject islands. Models are able to learn left branch islands and learn coordination islands gradiently, but fail to learn sentential subject islands. Through these controls and new tests, we provide evidence that model behavior is due to finer-grained expectations than gross syntactic complexity, but also that the models are conspicuously un-humanlike in some of their performance characteristics.

Left Node Blocking

The Left Elbow Constraint

Major Constituents and Two Dependency Grammar Constraints on Sharing in Coordination

Shared Material and Grammar: Toward A Dependency Grammar Theory of Non-Gapping Coordination for English and German

Bracketing paradoxes: a dependency grammar analysis in terms of morph catenae

Coherence : A Dependency Grammar Analysis

Bare Phrase Structure, Label-Less Trees, and Specifier-Less Syntax. is Minimalism Becoming a Dependency Grammar?

Coordination and Referential Dependencies: a Dependency Grammar Account in Terms of Predicate-Valent Structures

NPs, Not DPs: the NP Vs. DP Debate in the Context of Dependency Grammar

Scope Domains Toward a Dependency Grammar Account of the Syntactic Distribution of Negative Polarity Items

Constructions Are Catenae: Construction Grammar Meets Dependency Grammar

Comparative Coordination Vs. Comparative Subordination

Beyond the Constituent - A Dependency Grammar Analysis of Chains

The Role of Working Memory in Shaping Syntactic Dependency Structures

Catenae: Introducing a Novel Unit of Syntactic Analysis

Factors influencing dependency parsing of coordinating structure

Dependency networks: A discussion note about Lamb’s theory of linguistic structure

What Syntactic Structures block Dependencies in RNN Language Models?

Ellipsis in Dependency Grammar

Answer Fragments

Processing Coordination Ambiguity