Abstract:Deep Generative Models (DGMs) are able to extract high-level representations from massive unlabeled data and are explainable from a probabilistic perspective. Such characteristics favor sequence modeling tasks. However, it still remains a huge challenge to model sequences with DGMs. Unlike real-valued data that can be directly fed into models, sequence data consist of discrete elements and require being transformed into certain representations first. This leads to the following two challenges. First, high-level features are sensitive to small variations of inputs as well as the way of representing data. Second, the models are more likely to lose long-term information during multiple transformations. In this paper, we propose a Hierarchical Deep Generative Model With Dual Memory to address the two challenges. Furthermore, we provide a method to efficiently perform inference and learning on the model. The proposed model extends basic DGMs with an improved hierarchically organized multi-layer architecture. Besides, our model incorporates memories along dual directions, respectively denoted as broad memory and deep memory. The model is trained end-to-end by optimizing a variational lower bound on data log-likelihood using the improved stochastic variational method. We perform experiments on several tasks with various datasets and obtain excellent results. The results of language modeling show our method significantly outperforms state-of-the-art results in terms of generative performance. Extended experiments including document modeling and sentiment analysis, prove the high-effectiveness of dual memory mechanism and latent representations. Text random generation provides a straightforward perception for advantages of our model.

Sequence Modeling with Hierarchical Deep Generative Models with Dual Memory.

Learning to Generate with Memory

Hierarchical Topic Modeling with Nested Hierarchical Dirichlet Process

Characterizing A Database of Sequential Behaviors with Latent Dirichlet Hidden Markov Models

Learning Deep Generative Models with Doubly Stochastic Gradient MCMC

MemoryGAN: GAN Generator As Heterogeneous Memory for Compositional Image Synthesis

HVAE: A Deep Generative Model Via Hierarchical Variational Auto-Encoder for Multi-View Document Modeling

Learning Hierarchical Features from Generative Models

Generative Text Convolutional Neural Network for Hierarchical Document Representation Learning

Generative Modeling with Explicit Memory

Hierarchical Memory Learning for Fine-Grained Scene Graph Generation.

Building, Reusing, and Generalizing Abstract Representations from Concrete Sequences

A Semi-shared Hierarchical Joint Model for Sequence Labeling

Unified Generative and Discriminative Training for Multi-modal Large Language Models

Extending Memory for Language Modelling

Learning Hierarchical Structures On-The-Fly With A Recurrent-Recursive Model For Sequences

On Hierarchical Multi-Resolution Graph Generative Models

Hierarchically Gated Recurrent Neural Network for Sequence Modeling

A dual channel class hierarchy based recurrent language modeling

Hierarchical Multi-turn Dialogue Generation Model Based on Double-layer Decoding

Deep Generative Dual Memory Network for Continual Learning