Abstract:This thesis focuses on improving the pre-training of natural language models using unsupervised raw data to make them more efficient and aligned with downstream applications. In the first part, we introduce three alternative pre-training objectives to BERT's Masked Language Modeling (MLM), namely Random Token Substitution (RTS), Cluster-based Random Token Substitution (C-RTS), and Swapped Language Modeling (SLM). These objectives involve token swapping instead of masking, with RTS and C-RTS aiming to predict token originality and SLM predicting the original token values. Results show that RTS and C-RTS require less pre-training time while maintaining performance comparable to MLM. Surprisingly, SLM outperforms MLM on certain tasks despite using the same computational budget. In the second part, we proposes self-supervised pre-training tasks that align structurally with downstream applications, reducing the need for labeled data. We use large corpora like Wikipedia and CC-News to train models to recognize if text spans originate from the same paragraph or document in several ways. By doing continuous pre-training, starting from existing models like RoBERTa, ELECTRA, DeBERTa, BART, and T5, we demonstrate significant performance improvements in tasks like Fact Verification, Answer Sentence Selection, and Summarization. These improvements are especially pronounced when limited annotation data is available. The proposed objectives also achieve state-of-the-art results on various benchmark datasets, including FEVER (dev set), ASNQ, WikiQA, and TREC-QA, as well as enhancing the quality of summaries. Importantly, these techniques can be easily integrated with other methods without altering the internal structure of Transformer models, making them versatile for various NLP applications.

Robustly Optimized and Distilled Training for Natural Language Understanding

Multi-Task Learning in Natural Language Processing: An Overview

BAM! Born-Again Multi-Task Networks for Natural Language Understanding

Task-Attentive Transformer Architecture for Continual Learning of Vision-and-Language Tasks Using Knowledge Distillation

Multi-task learning for natural language processing in the 2020s: where are we going?

Pre-training Language Model As a Multi-perspective Course Learner.

Resonance RoPE: Improving Context Length Generalization of Large Language Models

XtremeDistil: Multi-stage Distillation for Massive Multilingual Models

How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classes

MT2KD: Towards A General-Purpose Encoder for Speech, Speaker, and Audio Events

Deepening Hidden Representations from Pre-trained Language Models for Natural Language Understanding

Enhancing Document Information Analysis with Multi-Task Pre-training: A Robust Approach for Information Extraction in Visually-Rich Documents

Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data

XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation

Multi-Task Learning for Front-End Text Processing in TTS

Co-Training Transformer for Remote Sensing Image Classification, Segmentation, and Detection

Structural Self-Supervised Objectives for Transformers

Making Small Language Models Better Multi-task Learners with Mixture-of-Task-Adapters

Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation

Efficient End-to-End Visual Document Understanding with Rationale Distillation