Abstract:Large-scale pre-trained language models (e.g., BERT) have attracted great attention in recent years. It is straightforward to fine-tune them on natural language understanding tasks such as text classification, however, effectively and efficiently incorporating them into natural language generation tasks such as neural machine translation remains a challenging problem. In this paper, we integrate two pre-trained BERT models from the source and target language domains into a sequence-to-sequence model by introducing light-weight adapter modules. The adapters are inserted between BERT layers and tuned on downstream tasks, while the parameters of BERT models are fixed during fine-tuning. As pre-trained language models are usually very deep, inserting adapters into all layers will result in a considerable scale of new parameters. To deal with this problem, we introduce latent variables to decide whether using adapters or not in each layer, which are learned during fine-tuning. In this way, the model is able to automatically determine which adapters to use, therefore hugely promoting the parameter efficiency and decoding speed. We evaluate the proposed framework on various neural machine translation tasks. Equipped with parallel sequence decoding, our model consistently outperforms autoregressive baselines while reducing the inference latency by half. With automatic adapter selection, the proposed model further achieves 20% speedup while still outperforming autoregressive baselines. When applied to autoregressive decoding, the proposed model can also achieve comparable performance with the state-of-the-art baseline models.

Meta-adapter: efficient cross-lingual adaptation with meta-learning

Exploiting Adapters for Cross-Lingual Low-Resource Speech Recognition

Multilingual Meta-Transfer Learning for Low-Resource Speech Recognition

Cross-lingual Adaption Model-Agnostic Meta-Learning for Natural Language Understanding

Meta Learning for End-to-End Low-Resource Speech Recognition

One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization

Efficient Adapter Finetuning for Tail Languages in Streaming Multilingual ASR

Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation

Efficient Adaptation: Enhancing Multilingual Models for Low-Resource Language Translation

Lightweight Adapter Tuning for Multilingual Speech Translation

AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Merging

Meta-Adapter: An Online Few-shot Learner for Vision-Language Model

Meta-Learning Adaptable Foundation Models

Language fusion via adapters for low-resource speech recognition

Language-Universal Adapter Learning with Knowledge Distillation for End-to-End Multilingual Speech Recognition

How to Learn a New Language? An Efficient Solution for Self-Supervised Learning Models Unseen Languages Adaption in Low-Resource Scenario

Adaptive Adapters: an Efficient Way to Incorporate BERT into Neural Machine Translation

Parameter-Efficient Fine-Tuning With Adapters

On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation

UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling

Good Meta-tasks Make A Better Cross-lingual Meta-transfer Learning for Low-resource Languages.