RoBERTa: A robustly optimized BERT pretraining approach (2019)

Liu Yinhan, Ott Myle, Goyal Naman, Du Jingfei, Joshi Mandar, Chen Danqi, Levy Omer, Lewis Mike

2019-01-01

Abstract:

What problem does this paper attempt to address?

Roberta: A robustly optimized bert pretraining approach

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov

2019-07-26

Abstract:
â€ œRoberta: A robustly optimized bert pretraining approach, â€

Yinhan Liu

2019-01-01

Abstract:
A robustly optimized bert pretraining approach [J]

Liu Y Roberta

2019-01-01

Abstract:
931 L. Zettlemoyer, and V. Stoyanov,“Roberta: A robustly optimized bert 932 pretraining approach,”

Y Liu, M Ott, N Goyal, J Du, M Joshi, D Chen, O Levy, M Lewis

2019-01-01

Abstract:
945 L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach

Y Liu, M Ott, N Goyal, J Du, M Joshi, D Chen, O Levy, M Lewis

Abstract:
A Robustly Optimized BERT Pre-training Approach with Post-training

Zhuang Liu,Wayne Lin,Ya Shi,Jun Zhao

DOI: https://doi.org/10.1007/978-3-030-84186-7_31

2021-01-01

Abstract:In the paper, we present a ‘pre-training’+‘post-training’+‘fine-tuning’ three-stage paradigm, which is a supplementary framework for the standard ‘pre-training’+‘fine-tuning’ language model approach. Furthermore, based on three-stage paradigm, we present a language model named PPBERT. Compared with original BERT architecture that is based on the standard twostage paradigm, we do not fine-tune pre-trained model directly, but rather post-train it on the domain or task related dataset first, which helps to better incorporate task-awareness knowledge and domain-awareness knowledge within pre-trained model, also from the training dataset reduce bias. Extensive experimental results indicate that proposed model improves the performance of the baselines on 24 NLP tasks, which includes eight GLUE benchmarks, eight SuperGLUE benchmarks, six extractive question answering benchmarks. More remarkably, our proposed model is a more flexible and pluggable model, where post-training approach is able to be plugged into other PLMs that are based on BERT. Extensive ablations further validate the effectiveness and its state-of-the-art (SOTA) performance. The open source code, pre-trained models and post-trained models are available publicly.
RoChBert: Towards Robust BERT Fine-tuning for Chinese

Zihan Zhang,Jinfeng Li,Ning Shi,Bo Yuan,Xiangyu Liu,Rong Zhang,Hui Xue,Donghong Sun,Chao Zhang

DOI: https://doi.org/10.48550/arXiv.2210.15944

2022-10-28

Computation and Language

Abstract:Despite of the superb performance on a wide range of tasks, pre-trained language models (e.g., BERT) have been proved vulnerable to adversarial texts. In this paper, we present RoChBERT, a framework to build more Robust BERT-based models by utilizing a more comprehensive adversarial graph to fuse Chinese phonetic and glyph features into pre-trained representations during fine-tuning. Inspired by curriculum learning, we further propose to augment the training dataset with adversarial texts in combination with intermediate samples. Extensive experiments demonstrate that RoChBERT outperforms previous methods in significant ways: (i) robust -- RoChBERT greatly improves the model robustness without sacrificing accuracy on benign texts. Specifically, the defense lowers the success rates of unlimited and limited attacks by 59.43% and 39.33% respectively, while remaining accuracy of 93.30%; (ii) flexible -- RoChBERT can easily extend to various language models to solve different downstream tasks with excellent performance; and (iii) efficient -- RoChBERT can be directly applied to the fine-tuning stage without pre-training language model from scratch, and the proposed data augmentation method is also low-cost.
StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding

Wei Wang,Bin Bi,Ming Yan,Chen Wu,Zuyi Bao,Jiangnan Xia,Liwei Peng,Luo Si

DOI: https://doi.org/10.48550/arXiv.1908.04577

2019-08-13

Computation and Language

Abstract:Recently, the pre-trained language model, BERT (and its robustly optimized version RoBERTa), has attracted a lot of attention in natural language understanding (NLU), and achieved state-of-the-art accuracy in various NLU tasks, such as sentiment classification, natural language inference, semantic textual similarity and question answering. Inspired by the linearization exploration work of Elman [8], we extend BERT to a new model, StructBERT, by incorporating language structures into pre-training. Specifically, we pre-train StructBERT with two auxiliary tasks to make the most of the sequential order of words and sentences, which leverage language structures at the word and sentence levels, respectively. As a result, the new model is adapted to different levels of language understanding required by downstream tasks. The StructBERT with structural pre-training gives surprisingly good empirical results on a variety of downstream tasks, including pushing the state-of-the-art on the GLUE benchmark to 89.0 (outperforming all published models), the F1 score on SQuAD v1.1 question answering to 93.0, the accuracy on SNLI to 91.7.
ROSITA: Refined BERT cOmpreSsion with InTegrAted techniques

Yuanxin Liu,Zheng Lin,Fengcheng Yuan

DOI: https://doi.org/10.48550/arXiv.2103.11367

2021-03-21

Abstract:Pre-trained language models of the BERT family have defined the state-of-the-arts in a wide range of NLP tasks. However, the performance of BERT-based models is mainly driven by the enormous amount of parameters, which hinders their application to resource-limited scenarios. Faced with this problem, recent studies have been attempting to compress BERT into a small-scale model. However, most previous work primarily focuses on a single kind of compression technique, and few attention has been paid to the combination of different methods. When BERT is compressed with integrated techniques, a critical question is how to design the entire compression framework to obtain the optimal performance. In response to this question, we integrate three kinds of compression methods (weight pruning, low-rank factorization and knowledge distillation (KD)) and explore a range of designs concerning model architecture, KD strategy, pruning frequency and learning rate schedule. We find that a careful choice of the designs is crucial to the performance of the compressed model. Based on the empirical findings, our best compressed model, dubbed Refined BERT cOmpreSsion with InTegrAted techniques (ROSITA), is $7.5 \times$ smaller than BERT while maintains $98.5\%$ of the performance on five tasks of the GLUE benchmark, outperforming the previous BERT compression methods with similar parameter budget. The code is available at <a class="link-external link-https" href="https://github.com/llyx97/Rosita" rel="external noopener nofollow">this https URL</a>.

Computation and Language
MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining

Jacob Portes,Alex Trott,Sam Havens,Daniel King,Abhinav Venigalla,Moin Nadeem,Nikhil Sardana,Daya Khudia,Jonathan Frankle

2024-01-17

Abstract:Although BERT-style encoder models are heavily used in NLP research, many researchers do not pretrain their own BERTs from scratch due to the high cost of training. In the past half-decade since BERT first rose to prominence, many advances have been made with other transformer architectures and training configurations that have yet to be systematically incorporated into BERT. Here, we introduce MosaicBERT, a BERT-style encoder architecture and training recipe that is empirically optimized for fast pretraining. This efficient architecture incorporates FlashAttention, Attention with Linear Biases (ALiBi), Gated Linear Units (GLU), a module to dynamically remove padded tokens, and low precision LayerNorm into the classic transformer encoder block. The training recipe includes a 30% masking ratio for the Masked Language Modeling (MLM) objective, bfloat16 precision, and vocabulary size optimized for GPU throughput, in addition to best-practices from RoBERTa and other encoder models. When pretrained from scratch on the C4 dataset, this base model achieves a downstream average GLUE (dev) score of 79.6 in 1.13 hours on 8 A100 80 GB GPUs at a cost of roughly $20. We plot extensive accuracy vs. pretraining speed Pareto curves and show that MosaicBERT base and large are consistently Pareto optimal when compared to a competitive BERT base and large. This empirical speed up in pretraining enables researchers and engineers to pretrain custom BERT-style models at low cost instead of finetune on existing generic models. We open source our model weights and code.

Computation and Language,Machine Learning
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Kenton Lee,Jacob Devlin,Ming-Wei Chang,Kristina Toutanova

DOI: https://doi.org/10.18653/v1/N19-1423

Abstract:We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).

Computer Science
ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT

Rui Pan,Shizhe Diao,Jianlin Chen,Tong Zhang

DOI: https://doi.org/10.48550/arXiv.2211.17201

2022-12-01

Abstract:In this paper, we present ExtremeBERT, a toolkit for accelerating and customizing BERT pretraining. Our goal is to provide an easy-to-use BERT pretraining toolkit for the research community and industry. Thus, the pretraining of popular language models on customized datasets is affordable with limited resources. Experiments show that, to achieve the same or better GLUE scores, the time cost of our toolkit is over $6\times$ times less for BERT Base and $9\times$ times less for BERT Large when compared with the original BERT paper. The documentation and code are released at <a class="link-external link-https" href="https://github.com/extreme-bert/extreme-bert" rel="external noopener nofollow">this https URL</a> under the Apache-2.0 license.

Computation and Language,Machine Learning,Optimization and Control
ARoBERT: An ASR Robust Pre-Trained Language Model for Spoken Language Understanding

Chengyu Wang,Suyang Dai,Yipeng Wang,Fei Yang,Minghui Qiu,Kehan Chen,Wei Zhou,Jun Huang

DOI: https://doi.org/10.1109/taslp.2022.3153268

2022-01-01

Abstract:Spoken Language Understanding (SLU) aims to interpret the meanings of human speeches in order to support various human-machine interaction systems. A key technique for SLU is Automatic Speech Recognition (ASR), which transcribes speech signals into text contents. As the output texts of modern ASR systems unavoidably contain errors, mainstream SLU models either trained or tested on texts transcribed by ASR systems would not be sufficiently error robust. We present ARoBERT, an ASR Robust BERT model, which can be fine-tuned to solve a variety of SLU tasks with noisy inputs. To guarantee the robustness of ARoBERT, during pretraining, we decrease the fluctuations of language representations when some parts of the input texts are replaced by homophones or synophones. Specifically, we propose two novel self-supervised pre-training tasks for ARoBERT, namely Phonetically-aware Masked Language Modeling (PMLM) and ASR Model-adaptive Masked Language Modeling (AMMLM). The PMLM task explicitly fuses the knowledge of word phonetic similarities into the pre-training process, which forces homophones and synophones to share similar representations. In AMMLM, a data-driven algorithm is further introduced to mine typical ASR errors such that ARoBERT can tolerate ASR model errors. In the experiments, we evaluate ARoBERT over multiple datasets. The results show the superiority of ARoBERT, which consistently outperforms strong baselines. We have also shown that ARoBERT outperforms state-of-the-arts on a public benchmark. Currently, ARoBERT has been deployed in an online production system with significant improvements.
A Win-win Deal: Towards Sparse and Robust Pre-trained Language Models

Yuanxin Liu,Fandong Meng,Zheng Lin,Jiangnan Li,Peng Fu,Yanan Cao,Weiping Wang,Jie Zhou

DOI: https://doi.org/10.48550/arxiv.2210.05211

2022-01-01

Abstract: Despite the remarkable success of pre-trained language models (PLMs), they still face two challenges: First, large-scale PLMs are inefficient in terms of memory footprint and computation. Second, on the downstream tasks, PLMs tend to rely on the dataset bias and struggle to generalize to out-of-distribution (OOD) data. In response to the efficiency problem, recent studies show that dense PLMs can be replaced with sparse subnetworks without hurting the performance. Such subnetworks can be found in three scenarios: 1) the fine-tuned PLMs, 2) the raw PLMs and then fine-tuned in isolation, and even inside 3) PLMs without any parameter fine-tuning. However, these results are only obtained in the in-distribution (ID) setting. In this paper, we extend the study on PLMs subnetworks to the OOD setting, investigating whether sparsity and robustness to dataset bias can be achieved simultaneously. To this end, we conduct extensive experiments with the pre-trained BERT model on three natural language understanding (NLU) tasks. Our results demonstrate that \textbf{sparse and robust subnetworks (SRNets) can consistently be found in BERT}, across the aforementioned three scenarios, using different training and compression methods. Furthermore, we explore the upper bound of SRNets using the OOD information and show that \textbf{there exist sparse and almost unbiased BERT subnetworks}. Finally, we present 1) an analytical study that provides insights on how to promote the efficiency of SRNets searching process and 2) a solution to improve subnetworks' performance at high sparsity. The code is available at https://github.com/llyx97/sparse-and-robust-PLM.
{E}fficient{BERT}: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation

Chenhe Dong,Guangrun Wang,Hang Xu,Jiefeng Peng,Xiaozhe Ren,Xiaodan Liang

DOI: https://doi.org/10.18653/v1/2021.findings-emnlp.123

2021-01-01

Abstract: Pre-trained language models have shown remarkable results on various NLP tasks. Nevertheless, due to their bulky size and slow inference speed, it is hard to deploy them on edge devices. In this paper, we have a critical insight that improving the feed-forward network (FFN) in BERT has a higher gain than improving the multi-head attention (MHA) since the computational cost of FFN is 2$\sim$3 times larger than MHA. Hence, to compact BERT, we are devoted to designing efficient FFN as opposed to previous works that pay attention to MHA. Since FFN comprises a multilayer perceptron (MLP) that is essential in BERT optimization, we further design a thorough search space towards an advanced MLP and perform a coarse-to-fine mechanism to search for an efficient BERT architecture. Moreover, to accelerate searching and enhance model transferability, we employ a novel warm-up knowledge distillation strategy at each search stage. Extensive experiments show our searched EfficientBERT is 6.9$\times$ smaller and 4.4$\times$ faster than BERT$\rm_{BASE}$, and has competitive performances on GLUE and SQuAD Benchmarks. Concretely, EfficientBERT attains a 77.7 average score on GLUE \emph{test}, 0.7 higher than MobileBERT$\rm_{TINY}$, and achieves an 85.3/74.5 F1 score on SQuAD v1.1/v2.0 \emph{dev}, 3.2/2.7 higher than TinyBERT$_4$ even without data augmentation. The code is released at https://github.com/cheneydon/efficient-bert.
Breaking MLPerf Training: A Case Study on Optimizing BERT

Yongdeok Kim,Jaehyung Ahn,Myeongwoo Kim,Changin Choi,Heejae Kim,Narankhuu Tuvshinjargal,Seungwon Lee,Yanzi Zhang,Yuan Pei,Xiongzhan Linghu,Jingkun Ma,Lin Chen,Yuehua Dai,Sungjoo Yoo

2024-02-04

Abstract:Speeding up the large-scale distributed training is challenging in that it requires improving various components of training including load balancing, communication, optimizers, etc. We present novel approaches for fast large-scale training of BERT model which individually ameliorates each component thereby leading to a new level of BERT training performance. Load balancing is imperative in distributed BERT training since its training datasets are characterized by samples with various lengths. Communication cost, which is proportional to the scale of distributed training, needs to be hidden by useful computation. In addition, the optimizers, e.g., ADAM, LAMB, etc., need to be carefully re-evaluated in the context of large-scale distributed training. We propose two new ideas, (1) local presorting based on dataset stratification for load balancing and (2) bucket-wise gradient clipping before allreduce which allows us to benefit from the overlap of gradient computation and synchronization as well as the fast training of gradient clipping before allreduce. We also re-evaluate existing optimizers via hyperparameter optimization and utilize ADAM, which also contributes to fast training via larger batches than existing methods. Our proposed methods, all combined, give the fastest MLPerf BERT training of 25.1 (22.3) seconds on 1,024 NVIDIA A100 GPUs, which is 1.33x (1.13x) and 1.57x faster than the other top two (one) submissions to MLPerf v1.1 (v2.0). Our implementation and evaluation results are available at MLPerf v1.1~v2.1.

Machine Learning,Computation and Language
EarlyBERT: Efficient BERT Training Via Early-bird Lottery Tickets

Xiaohan Chen,Yu Cheng,Shuohang Wang,Zhe Gan,Zhangyang Wang,Jingjing Liu

DOI: https://doi.org/10.18653/v1/2021.acl-long.171

2021-01-01

Abstract:Heavily overparameterized language models such as BERT, XLNet and T5 have achieved impressive success in many NLP tasks. However, their high model complexity requires enormous computation resources and extremely long training time for both pre-training and fine-tuning. Many works have studied model compression on large NLP models, but only focusing on reducing inference time while still requiring an expensive training process. Other works use extremely large batch sizes to shorten the pre-training time, at the expense of higher computational resource demands. In this paper, inspired by the Early-Bird Lottery Tickets recently studied for computer vision tasks, we propose EarlyBERT, a general computationally-efficient training algorithm applicable to both pre-training and fine-tuning of large-scale language models. By slimming the self-attention and fully-connected sub-layers inside a transformer, we are the first to identify structured winning tickets in the early stage of BERT training. We apply those tickets towards efficient BERT training, and conduct comprehensive pre-training and fine-tuning experiments on GLUE and SQuAD downstream tasks. Our results show that EarlyBERT achieves comparable performance to standard BERT, with 35~45% less training time. Code is available at https://github.com/VITA-Group/EarlyBERT.
GottBERT: a pure German Language Model

Raphael Scheible,Fabian Thomczyk,Patric Tippmann,Victor Jaravine,Martin Boeker

DOI: https://doi.org/10.48550/arXiv.2012.02110

2020-12-04

Abstract:Lately, pre-trained language models advanced the field of natural language processing (NLP). The introduction of Bidirectional Encoders for Transformers (BERT) and its optimized version RoBERTa have had significant impact and increased the relevance of pre-trained models. First, research in this field mainly started on English data followed by models trained with multilingual text corpora. However, current research shows that multilingual models are inferior to monolingual models. Currently, no German single language RoBERTa model is yet published, which we introduce in this work (GottBERT). The German portion of the OSCAR data set was used as text corpus. In an evaluation we compare its performance on the two Named Entity Recognition (NER) tasks Conll 2003 and GermEval 2014 as well as on the text classification tasks GermEval 2018 (fine and coarse) and GNAD with existing German single language BERT models and two multilingual ones. GottBERT was pre-trained related to the original RoBERTa model using fairseq. All downstream tasks were trained using hyperparameter presets taken from the benchmark of German BERT. The experiments were setup utilizing FARM. Performance was measured by the $F_{1}$ score. GottBERT was successfully pre-trained on a 256 core TPU pod using the RoBERTa BASE architecture. Even without extensive hyper-parameter optimization, in all NER and one text classification task, GottBERT already outperformed all other tested German and multilingual models. In order to support the German NLP field, we publish GottBERT under the AGPLv3 license.

Computation and Language,Machine Learning
BERTer: The Efficient One

Pradyumna Saligram,Andrew Lanpouthakoun

2024-07-19

Abstract:We explore advanced fine-tuning techniques to boost BERT's performance in sentiment analysis, paraphrase detection, and semantic textual similarity. Our approach leverages SMART regularization to combat overfitting, improves hyperparameter choices, employs a cross-embedding Siamese architecture for improved sentence embeddings, and introduces innovative early exiting methods. Our fine-tuning findings currently reveal substantial improvements in model efficiency and effectiveness when combining multiple fine-tuning architectures, achieving a state-of-the-art performance score of on the test set, surpassing current benchmarks and highlighting BERT's adaptability in multifaceted linguistic tasks.

Computation and Language,Machine Learning

RoBERTa: A robustly optimized BERT pretraining approach (2019)

Roberta: A robustly optimized bert pretraining approach

â€ œRoberta: A robustly optimized bert pretraining approach, â€

A robustly optimized bert pretraining approach [J]

931 L. Zettlemoyer, and V. Stoyanov,“Roberta: A robustly optimized bert 932 pretraining approach,”

945 L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach

A Robustly Optimized BERT Pre-training Approach with Post-training

RoChBert: Towards Robust BERT Fine-tuning for Chinese

StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding

ROSITA: Refined BERT cOmpreSsion with InTegrAted techniques

MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT

ARoBERT: An ASR Robust Pre-Trained Language Model for Spoken Language Understanding

A Win-win Deal: Towards Sparse and Robust Pre-trained Language Models

{E}fficient{BERT}: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation

Breaking MLPerf Training: A Case Study on Optimizing BERT

EarlyBERT: Efficient BERT Training Via Early-bird Lottery Tickets

GottBERT: a pure German Language Model

BERTer: The Efficient One