Abstract:For guiding natural language generation, many semantic-driven methods have been proposed. While clearly improving the performance of the end-to-end training task, these existing semantic-driven methods still have clear limitations: for example, (i) they only utilize shallow semantic signals (e.g., from topic models) with only a single stochastic hidden layer in their data generation process, which suffer easily from noise (especially adapted for short-text etc.) and lack of interpretation; (ii) they ignore the sentence order and document context, as they treat each document as a bag of sentences, and fail to capture the long-distance dependencies and global semantic meaning of a document. To overcome these problems, we propose a novel semantic-driven language modeling framework, which is a method to learn a Hierarchical Language Model and a Recurrent Conceptualization-enhanced Gamma Belief Network, simultaneously. For scalable inference, we develop the auto-encoding Variational Recurrent Inference, allowing efficient end-to-end training and simultaneously capturing global semantics from a text corpus. Especially, this article introduces concept information derived from high-quality lexical knowledge graph Probase, which leverages strong interpretability and anti-nose capability for the proposed model. Moreover, the proposed model captures not only intra-sentence word dependencies, but also temporal transitions between sentences and inter-sentence concept dependence. Experiments conducted on several NLP tasks validate the superiority of the proposed approach, which could effectively infer meaningful hierarchical concept structure of document and hierarchical multi-scale structures of sequences, even compared with latest state-of-the-art Transformer-based models.

Recurrent Hierarchical Topic-Guided RNN for Language Generation

A Hierarchical Neural Autoencoder for Paragraphs and Documents

Hierarchical Topic Modeling with Nested Hierarchical Dirichlet Process

Hierarchical Concept-Driven Language Model.

Hierarchical Recurrent Neural Network for Document Modeling.

Sentence Level Recurrent Topic Model: Letting Topics Speak for Themselves

Backward and Forward Language Modeling for Constrained Sentence Generation

Exploration of Tree-based Hierarchical Softmax for Recurrent Language Models

Utilizing Recurrent Neural Network for Topic Discovery in Short Text Scenarios

A Neural Generative Model for Joint Learning Topics and Topic-Specific Word Embeddings

Generative Text Convolutional Neural Network for Hierarchical Document Representation Learning

Backbone Language Modeling for Constrained Natural Language Generation.

Hierarchical Latent Semantic Mapping for Automated Topic Generation

Recurrent Memory Networks for Language Modeling

textTOvec: Deep Contextualized Neural Autoregressive Topic Models of Language with Distributed Compositional Prior

Recurrent Topic-Transition GAN for Visual Paragraph Generation

Context based Text-generation using LSTM networks

Tree-Structured Neural Machine for Linguistics-Aware Sentence Generation

Modeling Local Dependence in Natural Language with Multi-channel Recurrent Neural Networks

Neural Language Modeling by Jointly Learning Syntax and Lexicon.

Global context-dependent recurrent neural network language model with sparse feature learning