Abstract:Sequence-to-sequence (seq2seq) learning is a popular fashion for large-scale pretraining language models. However, the prior seq2seq pretraining models generally focus on reconstructive objectives on the decoder side and neglect the effect of encoder-side supervision, which we argue may lead to sub-optimal performance. To verify our hypothesis, we first empirically study the functionalities of the encoder and decoder in seq2seq pretrained language models, and find that the encoder takes an important but under-exploitation role than the decoder regarding the downstream performance and neuron activation. Therefore, we propose an encoding-enhanced seq2seq pretraining strategy, namely E2S2, which improves the seq2seq models via integrating more efficient self-supervised information into the encoders. Specifically, E2S2 adopts two self-supervised objectives on the encoder side from two aspects: 1) locally denoising the corrupted sentence (denoising objective); and 2) globally learning better sentence representations (contrastive objective). With the help of both objectives, the encoder can effectively distinguish the noise tokens and capture high-level (i.e., syntactic and semantic) knowledge, thus strengthening the ability of seq2seq model to accurately achieve the conditional generation. On a large diversity of downstream natural language understanding and generation tasks, E2S2 dominantly improves the performance of its powerful backbone models, e.g., BART and T5. For example, upon BART backbone, we achieve +1.1% averaged gain on the general language understanding evaluation (GLUE) benchmark and +1.75% F_0.5 score improvement on CoNLL2014 dataset. We also provide in-depth analyses to show the improvement stems from better linguistic representation. We hope that our work will foster future self-supervision research on seq2seq language model pretraining.

Denoising Based Sequence-to-Sequence Pre-training for Text Generation

DENOISPEECH: DENOISING TEXT TO SPEECH WITH FRAME-LEVEL NOISE MODELING

Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation

E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generation

Pretraining-Based Natural Language Generation for Text Summarization

Rethinking Denoised Auto-Encoding in Language Pre-Training.

Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise

A Semi-Supervised Approach for Low-Resourced Text Generation.

CAPT: Contrastive Pre-Training for LearningDenoised Sequence Representations

Multilingual Denoising Pre-training for Neural Machine Translation

ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation

Adversarial Subsequences for Unconditional Text Generation

GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator

Adversarial Sub-sequence for Text Generation.

Enhanced Seq2Seq Autoencoder via Contrastive Learning for Abstractive Text Summarization

Non-Autoregressive Text Generation with Pre-trained Language Models

PGSO: Prompt-based Generative Sequence Optimization Network for Aspect-based Sentiment Analysis

Learning to Encode Text As Human-Readable Summaries Using Generative Adversarial Networks.

Generating Text with Deep Reinforcement Learning

Denoising Prestack Random Noise with Deep Generative Prior

s2s-ft: Fine-Tuning Pretrained Transformer Encoders for Sequence-to-Sequence Learning