Abstract:This paper develops a general framework for learning interpretable data representation via Long Short-Term Memory (LSTM) recurrent neural networks over hierarchal graph structures. Instead of learning LSTM models over the pre-fixed structures, we propose to further learn the intermediate interpretable multi-level graph structures in a progressive and stochastic way from data during the LSTM network optimization. We thus call this model the structure-evolving LSTM. In particular, starting with an initial element-level graph representation where each node is a small data element, the structure-evolving LSTM gradually evolves the multi-level graph representations by stochastically merging the graph nodes with high compatibilities along the stacked LSTM layers. In each LSTM layer, we estimate the compatibility of two connected nodes from their corresponding LSTM gate outputs, which is used to generate a merging probability. The candidate graph structures are accordingly generated where the nodes are grouped into cliques with their merging probabilities. We then produce the new graph structure with a Metropolis-Hasting algorithm, which alleviates the risk of getting stuck in local optimums by stochastic sampling with an acceptance probability. Once a graph structure is accepted, a higher-level graph is then constructed by taking the partitioned cliques as its nodes. During the evolving process, representation becomes more abstracted in higher-levels where redundant information is filtered out, allowing more efficient propagation of long-range data dependencies. We evaluate the effectiveness of structure-evolving LSTM in the application of semantic object parsing and demonstrate its advantage over state-of-the-art LSTM models on standard benchmarks.

Learning Hierarchical Structures On-The-Fly With A Recurrent-Recursive Model For Sequences

When Are Tree Structures Necessary for Deep Learning of Representations?

A New Hierarchical Temporal Memory Based on Recurrent Learning Unit

On the Practical Ability of Recurrent Neural Networks to Recognize Hierarchical Languages

Hierarchical Attention Encoder Decoder

Learning Hierarchical Structures with Differentiable Nondeterministic Stacks

Hierarchical Conflict Propagation: Sequence Learning in a Recurrent Deep Neural Network

Tree2Tree Learning with Memory Unit

Finding Hierarchical Structure in Binary Sequences: Evidence from Lindenmayer Grammar Learning

Learning Hierarchical Information Flow with Recurrent Neural Modules

Tree Memory Networks for Modelling Long-term Temporal Dependencies

Multiresolution Transformer Networks: Recurrence is Not Essential for Modeling Hierarchical Structure

Hierarchical Adversarially Learned Inference

A Semi-shared Hierarchical Joint Model for Sequence Labeling

A Hierarchical Model with Recurrent Convolutional Neural Networks for Sequential Sentence Classification

Sliding Hierarchical Recurrent Neural Networks for Sequence Classification

Interpretable Structure-Evolving LSTM

Hierarchical Recurrent Neural Network for Document Modeling.

Focused Hierarchical RNNs for Conditional Sequence Processing

Interpretable Structure-aware Document Encoders with Hierarchical Attention

Hierarchically Gated Recurrent Neural Network for Sequence Modeling