Abstract:Recent breakthroughs in Natural Language Processing (NLP) have been driven by language models trained on a massive amount of plain text. While powerful, deriving supervision from textual resources is still an open question. For example, language model pretraining often neglects the rich, freely-available structures in textual data. In this thesis, we describe three lines of work that seek to improve the training and evaluation of neural models using naturally-occurring supervision. We first investigate self-supervised training losses to help enhance the performance of pretrained language models for various NLP tasks. Specifically, we alter the sentence prediction loss to make it better suited to other pretraining losses and more challenging to solve. We design an intermediate finetuning step that uses self-supervised training to promote models' ability in cross-task generalization. Then we describe methods to leverage the structures in Wikipedia and paraphrases. In particular, we propose training losses to exploit hyperlinks, article structures, and article category graphs for entity-, discourse-, entailment-related knowledge. We propose a framework that uses paraphrase pairs to disentangle semantics and syntax in sentence representations. We extend the framework for a novel generation task that controls the syntax of output text with a sentential exemplar. Lastly, we discuss our work on tailoring textual resources for establishing challenging evaluation tasks. We introduce three datasets by defining novel tasks using various fan-contributed websites, including a long-form data-to-text generation dataset, a screenplay summarization dataset, and a long-form story generation dataset. These datasets have unique characteristics offering challenges to future work in their respective task settings.

Neural Text Generation from Structured Data with Application to the Biography Domain

Incorporating Knowledge into Neural Network for Text Representation.

Neural Text Generation from Rich Semantic Representations

Learning to generate one-sentence biographies from Wikidata

Enhancing Neural Data-To-Text Generation Models with External Background Knowledge.

Order-Planning Neural Text Generation from Structured Data

Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation

Recent Advances in Neural Text Generation: A Task-Agnostic Survey

Neural Language Generation: Formulation, Methods, and Evaluation

Extending Neural Generative Conversational Model using External Knowledge Sources

Data-to-text Generation with Entity Modeling

Unsupervised Inflection Generation Using Neural Language Modeling

Neural Academic Paper Generation

Leveraging Natural Supervision for Language Representation Learning and Generation

Improving Neural Question Generation using Deep Linguistic Representation

Generating Full Length Wikipedia Biographies: The Impact of Gender Bias on the Retrieval-Based Generation of Women Biographies

Neural data-to-text generation with dynamic content planning

Neural paraphrasing by automatically crawled and aligned sentence pairs

Using Large Language Models for Zero-Shot Natural Language Generation from Knowledge Graphs

Collaborative Storytelling with Large-scale Neural Language Models

Neural Story Planning