Abstract:Human language is full of compositional syntactic structures, and although neural networks have contributed to groundbreaking improvements in computer systems that process language, widely-used neural network architectures still exhibit limitations in their ability to process syntax. To address this issue, prior work has proposed adding stack data structures to neural networks, drawing inspiration from theoretical connections between syntax and stacks. However, these methods employ deterministic stacks that are designed to track one parse at a time, whereas syntactic ambiguity, which requires a nondeterministic stack to parse, is extremely common in language. In this dissertation, we remedy this discrepancy by proposing a method of incorporating nondeterministic stacks into neural networks. We develop a differentiable data structure that efficiently simulates a nondeterministic pushdown automaton, representing an exponential number of computations with a dynamic programming algorithm. We incorporate this module into two predominant architectures: recurrent neural networks (RNNs) and transformers. We show that this raises their formal recognition power to arbitrary context-free languages, and also aids training, even on deterministic context-free languages. Empirically, neural networks with nondeterministic stacks learn context-free languages much more effectively than prior stack-augmented models, including a language with theoretically maximal parsing difficulty. We also show that an RNN augmented with a nondeterministic stack is capable of surprisingly powerful behavior, such as learning cross-serial dependencies, a well-known non-context-free pattern. We demonstrate improvements on natural language modeling and provide analysis on a syntactic generalization benchmark. This work represents an important step toward building systems that learn to use syntax in more human-like fashion.

Training with Exploration Improves a Greedy Stack-LSTM Parser

Transition-Based Dependency Parsing with Stack Long Short-Term Memory

Structured Training for Neural Network Transition-Based Parsing

A Neural Probabilistic Structured-Prediction Model for Transition-Based Dependency Parsing.

Learning Hierarchical Structures with Differentiable Nondeterministic Stacks

Incremental Parsing with Minimal Features Using Bi-Directional LSTM

Hybrid Oracle: Making Use of Ambiguity in Transition-based Chinese Dependency Parsing

AMR Parsing using Stack-LSTMs

Improving Autoregressive Training with Dynamic Oracles

Transition-Based Discourse Parsing with Multilayer Stack Long Short Term Memory

Exploiting limited data for parsing

A Neural Probabilistic Structured-Prediction Method for Transition-Based Natural Language Processing.

Nondeterministic Stacks in Neural Networks

Arc-Standard Spinal Parsing with Stack-LSTMs

Tackling Error Propagation through Reinforcement Learning: A Case of Greedy Dependency Parsing

Span-Based Constituency Parsing with a Structure-Label System and Provably Optimal Dynamic Oracles

Curriculum-Style Fine-Grained Adaption for Unsupervised Cross-Lingual Dependency Transfer

IncSQL: Training Incremental Text-to-SQL Parsers with Non-Deterministic Oracles

Shift-Reduce Task-Oriented Semantic Parsing with Stack-Transformers

Improved Discourse Parsing with Two-Step Neural Transition-Based Model

Learning latent variable grammars from complementary perspectives