Abstract:With the notable success of pretrained language models, the pretraining-fine-tuning paradigm has become a dominant solution for natural language understanding (NLU) tasks. Typically, the training instances of a target NLU task are introduced in a completely random order and treated equally at the fine-tuning stage. However, these instances can vary greatly in difficulty, and similar to human learning procedures, language models can benefit from an easy-to-difficult curriculum. Based on this concept, we propose a curriculum learning (CL) framework. Our framework consists of two stages, Review and Arrange, targeting the two main challenges in curriculum learning, i.e., how to define the difficulty of instances and how to arrange a curriculum based on the difficulty, respectively. In the first stage, we devise a cross-review (CR) method to train several teacher models first and then review the training set in a crossed manner to distinguish easy instances from difficult instances. In the second stage, two sampling algorithms, a coarse-grained arrangement (CGA) and a fine-grained arrangement (FGA), are proposed to arrange a curriculum for language models in which the learning materials start from the easiest instances, and more difficult instances are gradually added into the training procedure. Compared to previous heuristic CL methods, our framework can avoid the errors caused by a gap in difficulty between humans and machines and has strong generalization ability. We conduct comprehensive experiments, and the results show that our curriculum learning framework, without any manual model architecture design or use of external data, obtains significant and universal performance improvements on a wide range of NLU tasks in different languages.

Review-Based Curriculum Learning for Neural Machine Translation.

Reinforced Curriculum Learning on Pre-trained Neural Machine Translation Models

Meta-Curriculum Learning for Domain Adaptation in Neural Machine Translation

Review and Arrange: Curriculum Learning for Natural Language Understanding

An Empirical Exploration of Curriculum Learning for Neural Machine Translation

Self-Guided Curriculum Learning for Neural Machine Translation

Reinforcement Learning based Curriculum Optimization for Neural Machine Translation

Terminology-Enriched Meta-curriculum Learning for Domain Neural Machine Translation.

Token-wise Curriculum Learning for Neural Machine Translation

Norm-Based Curriculum Learning for Neural Machine Translation

Data Selection Curriculum for Neural Machine Translation

Continual Learning for Neural Machine Translation

Dynamic Curriculum Learning for Low-Resource Neural Machine Translation

Curriculum pre-training for stylized neural machine translation

Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation

ICL: Iterative Continual Learning for Multi-domain Neural Machine Translation

Curriculum Recommendations Using Transformer Base Model with InfoNCE Loss And Language Switching Method

A Review of the Evaluation System for Curriculum Learning

Revisiting Commonsense Reasoning in Machine Translation: Training, Evaluation and Challenge

Competence-based Curriculum Learning for Multilingual Machine Translation