Abstract:Pretrained language models have achieved great success in a wide range of natural language processing (NLP) problems, because they learn language representations from large-scale text corpora and can adapt to downstream tasks by finetuning them on annotated task data. However, such success relies on both large-scale text and annotated data, so the lack of training data is a major practical problem for many languages, especially low-resource languages. In this paper, we explore whether a pretrained English language model can benefit non-English NLP systems in low-resource scenarios, i.e., with limited text corpora or annotated data. To achieve this, we first propose cross-lingual knowledge transfer methods and then validate our methods in low-resource scenarios. Specifically, our cross-lingual knowledge transfer methods are applied in the training stages of language model pretraining or downstream finetuning. At the two stages, the methods are designed for the transfer of upstream general knowledge or downstream task-specific knowledge, respectively. In the experiments, we perform pretraining and finetuning with limited non-English data to simulate the low-resource scenarios. We evaluate our methods on ten downstream tasks over a wide range of languages, and present systematic comparisons among various knowledge transfer methods. Experimental results show that our methods successfully leverage a pretrained English language model to improve task performance in other languages. Besides, we demonstrate the multilinguality of the English language model in various application scenarios. Our findings imply the possibility to improve low-resource-language NLP systems with large-scale English language models.

Performance of Transfer Learning Model vs. Traditional Neural Network in Low System Resource Environment

From English To Foreign Languages: Transferring Pre-trained Language Models

Comparison of transfer-learning for lightweight pre-trained model on image classification

A new computationally efficient method to tune BERT networks – transfer learning

Low-Memory Neural Network Training: A Technical Report

Towards Compute-Optimal Transfer Learning

How to Determine the Most Powerful Pre-trained Language Model Without Brute Force Fine-tuning? an Empirical Survey

Achieving Pareto Optimality using Efficient Parameter Reduction for DNNs in Resource-Constrained Edge Environment

GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost

Performance of Convolution Neural Network based on Multiple GPUs with Different Data Communication Models

A Comparison of LSTM and BERT for Small Corpus

Can Pretrained English Language Models Benefit Non-English NLP Systems in Low-Resource Scenarios?

Optimising the Performance of Convolutional Neural Networks across Computing Systems using Transfer Learning

Comparison of Transfer Learning and Traditional Machine Learning Approach for Text Classification

How Transferable Are Neural Networks in NLP Applications?

Transfusion: Understanding Transfer Learning for Medical Imaging

Comparative Performance Analysis of Lightweight BERT-derived Models for Cybersecurity Corpus in Cloud Networks

A Systematic Performance Analysis of Deep Perceptual Loss Networks: Breaking Transfer Learning Conventions

A Challenging Benchmark for Low-Resource Learning

Exploiting Student Parallelism for Low-latency GPU Inference of BERT-like Models in Online Services

Gated Transfer Network for Transfer Learning