Abstract:Recent developments in unsupervised representation learning have successfully established the concept of transfer learning in NLP. Mainly three forces are driving the improvements in this area of research: More elaborated architectures are making better use of contextual information. Instead of simply plugging in static pre-trained representations, these are learned based on surrounding context in end-to-end trainable models with more intelligently designed language modelling objectives. Along with this, larger corpora are used as resources for pre-training large language models in a self-supervised fashion which are afterwards fine-tuned on supervised tasks. Advances in parallel computing as well as in cloud computing, made it possible to train these models with growing capacities in the same or even in shorter time than previously established models. These three developments agglomerate in new state-of-the-art (SOTA) results being revealed in a higher and higher frequency. It is not always obvious where these improvements originate from, as it is not possible to completely disentangle the contributions of the three driving forces. We set ourselves to providing a clear and concise overview on several large pre-trained language models, which achieved SOTA results in the last two years, with respect to their use of new architectures and resources. We want to clarify for the reader where the differences between the models are and we furthermore attempt to gain some insight into the single contributions of lexical/computational improvements as well as of architectural changes. We explicitly do not intend to quantify these contributions, but rather see our work as an overview in order to identify potential starting points for benchmark comparisons. Furthermore, we tentatively want to point at potential possibilities for improvement in the field of open-sourcing and reproducible research.

Pre-trained Language Model Representations for Language Generation

Pre-training Universal Language Representation

Language Models not just for Pre-training: Fast Online Neural Noisy Channel Modeling

Unified Language Model Pre-training for Natural Language Understanding and Generation

Efficient Adaptation of Pretrained Transformers for Abstractive Summarization

Pretraining-Based Natural Language Generation for Text Summarization

Text Summarization with Pretrained Encoders

Language Models are Good Translators

Improving Neural Machine Translation with Pre-trained Representation

Combining Temporal Event Relations and Pre-Trained Language Models for Text Summarization

Discovering Representation Sprachbund for Multilingual Pre-Training

Incorporating Pre-trained Model into Neural Machine Translation

Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization

Sequence Generation with Mixed Representations.

Sharing Pre-Trained Bert Decoder For A Hybrid Summarization

Language-Independent Representations Improve Zero-Shot Summarization

Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations

Improving Language Understanding by Generative Pre-Training

On the comparability of Pre-trained Language Models

Extrapolating Multilingual Understanding Models as Multilingual Generators

Sequence-to-Sequence Spanish Pre-trained Language Models