Abstract:In this thesis, we explore the use of deep neural networks for generation of natural language. Specifically, we implement two sequence-to-sequence neural variational models - variational autoencoders (VAE) and variational encoder-decoders (VED). VAEs for text generation are difficult to train due to issues associated with the Kullback-Leibler (KL) divergence term of the loss function vanishing to zero. We successfully train VAEs by implementing optimization heuristics such as KL weight annealing and word dropout. We also demonstrate the effectiveness of this continuous latent space through experiments such as random sampling, linear interpolation and sampling from the neighborhood of the input. We argue that if VAEs are not designed appropriately, it may lead to bypassing connections which results in the latent space being ignored during training. We show experimentally with the example of decoder hidden state initialization that such bypassing connections degrade the VAE into a deterministic model, thereby reducing the diversity of generated sentences. We discover that the traditional attention mechanism used in sequence-to-sequence VED models serves as a bypassing connection, thereby deteriorating the model's latent space. In order to circumvent this issue, we propose the variational attention mechanism where the attention context vector is modeled as a random variable that can be sampled from a distribution. We show empirically using automatic evaluation metrics, namely entropy and distinct measures, that our variational attention model generates more diverse output sentences than the deterministic attention model. A qualitative analysis with human evaluation study proves that our model simultaneously produces sentences that are of high quality and equally fluent as the ones generated by the deterministic attention counterpart.

A Deep Ensemble Model with Slot Alignment for Sequence-to-Sequence Natural Language Generation

Deep Reinforcement Learning for Dialogue Generation

Adversarial Learning for Neural Dialogue Generation.

A Persona-Based Neural Conversation Model

A Hierarchical Neural Autoencoder for Paragraphs and Documents

Deep Adversarial Learning for NLP.

Semi-Supervised Neural Text Generation by Joint Learning of Natural Language Generation and Natural Language Understanding Models

Neural Language Generation: Formulation, Methods, and Evaluation

EnsembleGAN: Adversarial Learning for Retrieval-Generation Ensemble Model on Short-Text Conversation

AUGNLG: Few-shot Natural Language Generation using Self-trained Data Augmentation

Natural Language Generation Using Reinforcement Learning with External Rewards

Natural Language Generation Using Sequential Models: A Survey

A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues

A Primer on Seq2Seq Models for Generative Chatbots

A Simple yet Efficient Ensemble Approach for AI-generated Text Detection

Generative Cooperative Networks for Natural Language Generation

Learning Evaluation Models from Large Language Models for Sequence Generation

Tree-Structured Neural Machine for Linguistics-Aware Sentence Generation

Boosting Naturalness of Language in Task-oriented Dialogues via Adversarial Training

Natural Language Generation with Neural Variational Models