Abstract:BERT (Bidirectional Encoder Representations from Transformers) has revolutionized the field of natural language processing through its exceptional performance on numerous tasks. Yet, the majority of researchers have mainly concentrated on enhancements related to the model structure, such as relative position embedding and more efficient attention mechanisms. Others have delved into pretraining tricks associated with Masked Language Modeling, including whole word masking. DeBERTa introduced an enhanced decoder adapted for BERT's encoder model for pretraining, proving to be highly effective. We argue that the design and research around enhanced masked language modeling decoders have been underappreciated. In this paper, we propose several designs of enhanced decoders and introduce BPDec (BERT Pretraining Decoder), a novel method for modeling training. Typically, a pretrained BERT model is fine-tuned for specific Natural Language Understanding (NLU) tasks. In our approach, we utilize the original BERT model as the encoder, making only changes to the decoder without altering the encoder. This approach does not necessitate extensive modifications to the encoder architecture and can be seamlessly integrated into existing fine-tuning pipelines and services, offering an efficient and effective enhancement strategy. Compared to other methods, while we also incur a moderate training cost for the decoder during the pretraining process, our approach does not introduce additional training costs during the fine-tuning phase. We test multiple enhanced decoder structures after pretraining and evaluate their performance on the GLUE tasks and SQuAD tasks. Our results demonstrate that BPDec, having only undergone subtle refinements to the model structure during pretraining, significantly enhances model performance without escalating the finetuning cost, inference time and serving budget.

Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

Multilingual Denoising Pre-training for Neural Machine Translation

Denoising Based Sequence-to-Sequence Pre-training for Text Generation

Rethinking Denoised Auto-Encoding in Language Pre-Training.

DENOISPEECH: DENOISING TEXT TO SPEECH WITH FRAME-LEVEL NOISE MODELING

BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese

CAPT: Contrastive Pre-Training for LearningDenoised Sequence Representations

GreekBART: The First Pretrained Greek Sequence-to-Sequence Model

BARThez: a Skilled Pretrained French Sequence-to-Sequence Model

BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model

Towards Transfer Learning for End-to-End Speech Synthesis from Deep Pre-Trained Language Models.

Domain Specific Fine-tuning of Denoising Sequence-to-Sequence Models for Natural Language Summarization

BARTScore: Evaluating Generated Text as Text Generation

IndicBART: A Pre-trained Model for Indic Natural Language Generation

Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation?

Unified Pre-training for Program Understanding and Generation

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Sentence Bottleneck Autoencoders from Transformer Language Models

BPDec: Unveiling the Potential of Masked Language Modeling Decoder in BERT pretraining

MASS: Masked Sequence to Sequence Pre-training for Language Generation