Abstract:How to properly model graphs is a long-existing and important problem in NLP area, where several popular types of graphs are knowledge graphs, semantic graphs and dependency graphs. Comparing with other data structures, such as sequences and trees, graphs are generally more powerful in representing complex correlations among entities. For example, a knowledge graph stores real-word entities (such as "Barack_Obama" and "U.S.") and their relations (such as "live_in" and "lead_by"). Properly encoding a knowledge graph is beneficial to user applications, such as question answering and knowledge discovery. Modeling graphs is also very challenging, probably because graphs usually contain massive and cyclic relations. Recent years have witnessed the success of deep learning, especially RNN-based models, on many NLP problems. Besides, RNNs and their variations have been extensively studied on several graph problems and showed preliminary successes. Despite the successes that have been achieved, RNN-based models suffer from several major drawbacks on graphs. First, they can only consume sequential data, thus linearization is required to serialize input graphs, resulting in the loss of important structural information. Second, the serialization results are usually very long, so it takes a long time for RNNs to encode them. In this thesis, we propose a novel graph neural network, named graph recurrent network (GRN). We study our GRN model on 4 very different tasks, such as machine reading comprehension, relation extraction and machine translation. Some take undirected graphs without edge labels, while the others have directed ones with edge labels. To consider these important differences, we gradually enhance our GRN model, such as further considering edge labels and adding an RNN decoder. Carefully designed experiments show the effectiveness of GRN on all these tasks.

Riemannian metrics for neural networks II: recurrent networks and learning symbolic data sequences

Residual Recurrent Neural Networks for Learning Sequential Representations.

Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations

Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics

A Critical Review of Recurrent Neural Networks for Sequence Learning

Learning Longer Memory in Recurrent Neural Networks

Recurrent neural networks: vanishing and exploding gradients are not the end of the story

A singular Riemannian Geometry Approach to Deep Neural Networks III. Piecewise Differentiable Layers and Random Walks on $n$-dimensional Classes

Hierarchically Gated Recurrent Neural Network for Sequence Modeling

Colorless green recurrent networks dream hierarchically

Lyapunov-Guided Representation of Recurrent Neural Network Performance

On the difficulty of learning chaotic dynamics with RNNs

Identifying nonlinear dynamical systems with multiple time scales and long-range dependencies

RotRNN: Modelling Long Sequences with Rotations

Tackling Graphical NLP problems with Graph Recurrent Networks

A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithms

Prediction of Time Series Gene Expression and Structural Analysis of Gene Regulatory Networks Using Recurrent Neural Networks

Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets

Scalable Stochastic Gradient Riemannian Langevin Dynamics in Non-Diagonal Metrics

Structural-RNN: Deep Learning on Spatio-Temporal Graphs