Abstract:Recent breakthroughs in Natural Language Processing (NLP) have been driven by language models trained on a massive amount of plain text. While powerful, deriving supervision from textual resources is still an open question. For example, language model pretraining often neglects the rich, freely-available structures in textual data. In this thesis, we describe three lines of work that seek to improve the training and evaluation of neural models using naturally-occurring supervision. We first investigate self-supervised training losses to help enhance the performance of pretrained language models for various NLP tasks. Specifically, we alter the sentence prediction loss to make it better suited to other pretraining losses and more challenging to solve. We design an intermediate finetuning step that uses self-supervised training to promote models' ability in cross-task generalization. Then we describe methods to leverage the structures in Wikipedia and paraphrases. In particular, we propose training losses to exploit hyperlinks, article structures, and article category graphs for entity-, discourse-, entailment-related knowledge. We propose a framework that uses paraphrase pairs to disentangle semantics and syntax in sentence representations. We extend the framework for a novel generation task that controls the syntax of output text with a sentential exemplar. Lastly, we discuss our work on tailoring textual resources for establishing challenging evaluation tasks. We introduce three datasets by defining novel tasks using various fan-contributed websites, including a long-form data-to-text generation dataset, a screenplay summarization dataset, and a long-form story generation dataset. These datasets have unique characteristics offering challenges to future work in their respective task settings.

Application of self-supervised learning in natural language processing

Application of Deep Learning-Based Natural Language Processing in Multilingual Sentiment Analysis

Self-Supervised Representation Learning: Introduction, advances, and challenges

Semi-Unsupervised Lifelong Learning for Sentiment Classification: Less Manual Data Annotation and More Self-Studying

Rethinking Semi-supervised Learning with Language Models

Semi-supervised Classification for Natural Language Processing

Pseudo Labeling Methods for Semi-Supervised Semantic Segmentation: A Review and Future Perspectives

Leveraging Natural Supervision for Language Representation Learning and Generation

Self-supervised Learning: A Succinct Review

Predictions For Pre-training Language Models

A Comprehensive Survey of Deep Learning Techniques Natural Language Processing

Learning How to Self-Learn: Enhancing Self-Training Using Neural Reinforcement Learning

Unsupervised Self-Training for Sentiment Analysis of Code-Switched Data

Improving BERT with Self-Supervised Attention

Pre-Trained Language Models and Their Applications

Bring Your Own Data! Self-Supervised Evaluation for Large Language Models

Self-Supervised Multimodal Learning: A Survey

A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends

Deep Learning and Its Applications to Natural Language Processing

Unsupervised Pre-training for Natural Language Generation: A Literature Review

An Overview of Deep Semi-Supervised Learning